Two men — Philip E. Thomas and George Brown — were the pioneers of the railroad. They spent the year 1826 investigating railway enterprises in England, which were at that time being tested in a comprehensive fashion as commercial ventures. Their investigation completed, they held an organizational meeting on February 12, 1827, including about twenty-five citizens, most of whom were Baltimore merchants or bankers. Chapter 123 of the 1826 Session Laws of Maryland, passed February 28, 1827, and the Commonwealth of Virginia on March 8, 1827, chartered the Baltimore and Ohio Rail Road Company, with the task of building a railroad from the port of Baltimore, Maryland west to a suitable point on the Ohio River.

The railroad, formally incorporated April 24, was intended to provide not only an alternative to, but also a faster route for Midwestern goods to reach the East Coast than the seven-year-old, hugely successful, but slow Erie Canal across upstate New York. Thomas was elected as the first president and Brown the treasurer. The capital of the proposed company was fixed at five million dollars.

Construction began on July 4, 1828, and the first section, from Baltimore west to Ellicott's Mills (now known as Ellicott City), opened two years later.

It was decided to follow the Patapsco River to a point near Parr's Ridge where the railroad would cross the fall line and descend into the valley of the Monocacy and Potomac Rivers. Further extensions opened to Frederick (including the short Frederick Branch) December 1, 1831, Point of Rocks April 2, 1832, Sandy Hook December 1, 1834 (the connection to the Winchester and Potomac Railroad at Harpers Ferry opening in 1837), Martinsburg May 1842, Hancock June 1842, Cumberland November 5, 1842, Piedmont July 21, 1851, Fairmont June 22, 1852, and its terminus at Wheeling, West Virginia (then part of Virginia).

The state of Maryland granted the B&O a charter to build a line from Baltimore to Washington, DC, in 1831, and the Washington Branch was opened in 1835. This line joined to the original mainline at Relay, Maryland, crossing the Patapsco on the Thomas Viaduct, which remains one of the B&O's signature structures. This line was partially funded by the state, and was operated separately for about 40 years, with the state taking a 25% cut of gross passenger receipts. This line was built in stone, much like the original mainline; by this time, however, strap rail was no longer used for new construction. Most of the stone bridges on the Old Main Line did not last long, being washed out by the periodic flooding of the Patapsco River and replaced at first by Bollman Truss Bridges.

The Annapolis and Elk Ridge Railroad to Annapolis eventually connected to this line at Annapolis Junction, Maryland. As an unwritten condition for the charter, it was understood that the state would not charter any competing line between Baltimore and Washington.

A steel and stone bridge was built across the Ohio River between Bellaire and Wheeling in 1871, connecting the B&O to the Central Ohio Railroad which it had leased starting in 1866. This provided a direct rail connection to Columbus, Ohio, and the lease marked the beginning of a series of expansions to the west and north.

Other railroads included in the B&O were:

• Winchester and Potomac Railroad and Winchester and Strasburg Railroad. This pair of lines connected with the B&O at Harper's Ferry, West Virginia, and constituted the only significant B&O trackage in present day Virginia.
• Sandusky, Mansfield and Newark Railroad leased through the Central Ohio.
• Pittsburgh and Connellsville Railroad. This was the B&O entry into Pittsburgh, thwarting the denial of a Pennsylvania charter to the B&O.
• Somerset and Cambria Railroad.
• Buffalo Railroad.
• West Virginia and Pittsburgh Railroad.
• Columbus and Cincinnati Midland Railroad leased through Central Ohio.

• Monongahela River Railroad.
• Marietta and Cincinnati Railroad. This was initially renamed as the Cincinnati, Washington and Baltimore Railroad in and then again to the Baltimore and Ohio Southwestern Railroad in 1889. The B&OSW absorbed the Ohio and Mississippi Railroad in 1893, giving the B&O a connection to St. Louis, Missouri, and finally the B&OSW disappeared into the rest of the system.
• Ohio River Railroad.
• Pittsburgh Junction Railroad.
• Pittsburgh and Western Railroad. This was originally a narrow gauge system which was standard gauged from 1883 to 1911. It formed the main B&O line west from Pittsburgh. The line passed the Mars Train Station in Mars, Pennsylvania, northwest of Pittsburgh.

• Cleveland Terminal and Valley Railroad. This was the B&O's entry into Cleveland, Ohio.
• Cleveland, Lorain and Wheeling Railroad.
• Chicago Terminal Transfer Railroad, reorganized in 1910 as the Baltimore and Ohio Chicago Terminal Railroad. This switching line was always operated as a separate company.
• Salisbury Railroad near Pittsburgh.
• Cincinnati, Hamilton and Dayton Railroad.
• Morgan and Kingwood Railroad.
• Coal and Coke Railroad.
• Cincinnati, Indianapolis and Western Railroad. This was originally part of the Cincinnati, Hamilton and Dayton, and gave the B&O a connection to Springfield, Illinois.

• Buffalo, Rochester and Pittsburgh Railway. This gave the B&O a line into New York state.
• Buffalo and Susquehanna Railroad. Part of the line was severed from the rest of the system by flooding, and became part of the Wellsville, Addison and Galeton Railroad in 1955.
• The Chicago and Alton Railroad was purchased by the B&O in 1931 and renamed the Alton Railroad. It was always operated separately and was eventually bought by the Gulf, Mobile and Ohio Railroad after receivership in 1942.

On July 20, 1877 there were bloody riots in Baltimore, Maryland, from Baltimore and Ohio Railroad workers. Nine rail workers were killed at the hands of the Maryland militia. The next day workers in Pittsburgh, Pennsylvania, staged a sympathy strike that was also met with an assault by the state militia; Pittsburgh then erupted into widespread rioting.

In 1866 the B&O began constructing the Metropolitan Branch west out of Washington, and was completed in 1873 after years of erratic effort. Before this line was laid, rail traffic west of Washington had to travel first to Relay or Baltimore before joining the main line.

The line cut a more or less straight line from Washington to Point of Rocks, Maryland, with many grades and large bridges. Upon the opening of this line, through passenger traffic was rerouted through Washington, and the old main line from Point of Rocks to Relay was reduced to secondary status as far as passenger service was concerned. Rebuilding after the turn of the century and double tracking in 1928 increased capacity; the "branches" became the de facto mainline, though the Old Main Line was retained as relief route.

Meanwhile the Pennsylvania Railroad (PRR) outmaneuvered the B&O to acquire the B&O's northern connection, the Philadelphia, Wilmington and Baltimore Railroad, cutting off the B&O's access to Philadelphia, Pennsylvania and New York.

The state of Maryland had stayed true to its implicit promise not to grant competing charters for the Baltimore/Washington line, but when a charter was granted to build a line from Baltimore to Pope's Creek in southern Maryland, lawyers for the Pennsylvania RR picked up on a clause in the unfulfilled charter allowing branches up to twenty miles long, from any point and in any direction. The projected route, passing through what is now Bowie, Maryland, could have a "branch" constructed that would allow service into Washington. The Pennsylvania picked up the charter through the agency of the Baltimore and Potomac Railroad and in 1872 service between Baltimore and Washington began. At the same time the PRR outmaneuvered the B&O and took control of the Long Bridge, B&O's connections to southern lines.

In response, the B&O chartered the Philadelphia Branch in Maryland and the Baltimore and Philadelphia Railroad in Delaware and Pennsylvania and built a parallel route, finished in 1886. The Baltimore Belt Line, opened in 1895, connected the main line to the Philadelphia Branch without the need for a car ferry across the Patapsco River, but the cost of constructing the Howard Street Tunnel drove the B&O to bankruptcy in 1896. Two other lines were built in attempts to reconnect to the south. The Alexandria Branch was built in 1874, starting from Hyattsville, Maryland, and ending at a ferry operation at Shepherd's Landing.

The Ferry operation continued until 1901 when the trackage rights agreement concluded as part of the construction of Washington Union Station saw the south end of the branch realigned to link to the PRR trackage in Anacostia, across the Anacostia River, into the Capitol Hill Tunnel, through Southwest Washington, D.C. to Potomac Yard in Alexandria, Virginia. The Alexandria Branch trackage to Shepherd's Landing was heavily used during World War II when traffic congestion on the Long Bridge caused the U.S. Army Corps of Engineers to construct a bridge along the original plan of the B&O: Alexandria to Shepherd's Landing, Washington. Trains of empty freight cars were routed north and south over the structure, which was demolished after the end of World War II.

Before either connection was made, however, another branch was built around the west side of Washington. Also,the B&O had organized a group of bankrupt railroads in Virginia into the Virginia Midland Railroad. The VM track rack from Alexandria, Virginia, to Danville, Virginia. The line projected west across the Potomac River was intended to cross the Potomac just north of the D.C. line, to continue southwest to a connection with the B&O-controlled Virginia Midland (VM) in Fairfax, Virginia (now Fairfax Station, Virginia, to distinguish it from what was Fairfax Court House, Virginia, and is now the City of Fairfax, Virginia), and if possible to a connection with the Richmond, Fredericksburg, and Potomac Railroad in Quantico, Virginia.

The branch was started in 1892 and reached Chevy Chase, Maryland, the same year. Financial problems in both the VM and B&O forced a halt to construction and led to the B&O's loss of control of the VM. Following bankruptcy, and control by the Pennsylvania Railroad, by the time the line was completed there was no longer any point to the river crossing. Thus the renamed Georgetown Branch came to serve a wide range of customers in Maryland and in Georgetown, such as the Potomac Electric & Power Co., the Washington Milling Company, and the U.S. Government. The line cut directly across the various creeks, and as a result required a short tunnel and what was said to be the longest wood trestle on the railroad over Rock Creek. The line was almost completely abandoned in 1986 by CSX and is presently used in part as the right-of-way for the Capital Crescent Trail.

Following its emergence from bankruptcy, control of the B&O was acquired by the Pennsylvania Railroad in 1901. A rising young PRR Vice President, Leonor F. Loree, was appointed President. Loree shared the Pennsy management's belief in infrastructure and the B&O at that time needed some of that. New classes of engines were built to haul longer, heavier trains faster. The Old Main Line was reworked, sections of the original right-of-way cut off by the straightening of curves and replacement of old, weight-restricted bridges with newer, heavier bridges. Most of Loree's work on the B&O physical plant remains evident today. Many iron and steel bridges on the railroad were replaced with stone (Pennsy preferred stone to the preference of the Reading and Lackawanna Railroad for concrete).

Chesapeake and Ohio Railroad took control of the B&O in 1963, and incorporated it, along with the Western Maryland Railway, into the Chessie System in 1973.Seven years later, the Chessie System merged with the Seaboard System Railroad to create CSX. In 1986, the B&O finally went out of corporate existence when it formally merged with the C&O (which itself formally merged with CSX later that same year).

At the height of railroading's golden age, the B&O was one of several trunk lines uniting the northeast quadrant of the United States into an industrial zone. It marked the southern border and corresponded to the New York Central's marking of the northern border.

The Pennsylvania Railroad controlled the center and smaller roads like the Lackawanna, Lehigh Valley, and the Erie survived largely through the Interstate Commerce Commission. The corners of this map are Baltimore in the southeast, Boston in the northeast, Chicago in the northwest, and St. Louis in the southwest.

When construction began on the B&O, railroad engineering was in its infancy. Unsure exactly which materials would suffice, the B&O erred on the side of sturdiness and built many of its early structures of granite. Even the track bed to which iron strap rail was affixed consisted of the stone.

Though the granite soon proved too unforgiving and expensive for track, most of the B&O's bridges have survived until the present, and many are still in active railroad use by CSX. Baltimore's Carrollton Viaduct, named in honor of Charles Carroll of Carrollton (who laid the cornerstone), was the B&O's first bridge, and is the world's oldest railroad bridge still in use. The Thomas Viaduct in Relay, Maryland, was the longest bridge in the United States upon its completion in 1835, and remains in use as well. The B&O made extensive use of the Bollman iron truss bridge in the mid 19th century; its durability and ease of assembly aided faster railroad construction.

As the B&O built west from Baltimore in its infancy, it followed the banks of the Patapsco River upstream to the water's source at Parrs Spring near present-day Mount Airy, Maryland. At the time little data about the operation of steam locomotives was available, and consequently the B&O was uncertain if metal wheels would grip the metal rails sufficiently to pull a train up to the top of Parrs Ridge. The railroad decided to construct two inclined planes on each side of the ridge along which teams of horses, and perhaps steam-powered winches, would assist pulling the trains uphill. The planes, about a mile long on each side of the ridge, quickly proved an operational bottleneck, and before the decade ended the B&O built a 5.5 mile long alternate route that became known as the Mount Airy Loop. The planes were quickly abandoned and forgotten, though some artifacts survive to the present.

The Mount Airy Branch is the surviving, in-use portion of the 1839-opened Mount Airy Loop. The Loop had been mainline track until superseded by the Mount Airy Cutoff and Tunnel in 1902.

The Frederick Branch was built because the city of Frederick would not pay the B&O the cost of routing the railroad through the rougher terrain into downtown Frederick. The branch opened on December 1, 1831. The continuation of the main line from Frederick Junction opened April 2, 1832.

The Patuxent Branch was constructed and split off from the Washington Branch at Savage, Maryland to serve a mill, a quarry, and other small industry. After 1925, the line was gradually cut back, and disconnected completely in 2005.

Originally intended as an extension of the railroad to a crossing of the Potomac River near the Chain Bridge, the agreement between the Pennsylvania Railroad and the B&O resulting from the rerouting of track for the Washington Union Station project put an end to the crossing and the branch settled down to being just a country railroad until the Washington, D.C. suburbs grew around it.

The B&O had decided against a direct line to Hagerstown, though the City had petitioned the Directors. Several north-south routes like the Cumberland Valley built through Hagerstown and the construction of the Western Maryland Railway to that city persuaded the B&O management to build a branch. It was decided that the branch would leave the mainline at Weaverton and wind its way through the hills of Western Maryland to Hagerstown. A station was constructed at the stub end of the line in downtown Hagerstown.