Wikipedia: Grey's Raid:
In response to the threat to Newport, General Sir Henry Clinton ordered 4,000 men under General Charles Grey to prepare for transport to Rhode Island while Admiral Lord Richard Howe sailed from New York to oppose d'Estaing.... By the time they arrived at Newport on September 1, the Americans had not only been put on the defensive, but had retreated from the island after the inconclusive August 29 Battle of Rhode Island.
Rather than disembark Grey's troops at Newport, Clinton decided to pursue other objectives.... He ordered the fleet to sail to New London, Connecticut, a potential raiding site. Finding too few ships there to bother with a landing, Clinton ordered Grey to "proceed without loss of time to the eastward" to raid New Bedford and Fairhaven on the Massachusetts mainland, and the island of Martha's Vineyard.
Early on September 4, Grey's fleet, led by the Royal Navy frigate Carysfort under Captain Robert Fanshawe, sailed for Buzzard's Bay. En route they encountered Lord Howe's fleet; the earl agreed to remain near Block Island until the raids were completed. Arriving in Buzzard's Bay that afternoon, the Carysfort had the misfortune to twice strike rocks; fortunately, neither incident was serious, and the fleet proceeded up the Acushnet River toward New Bedford and Fairhaven. That evening, Grey landed his troops at Clark's Point on the west bank of the river. They spent the night and the next morning destroying vessels, warehouses, and wharves "in the whole Extent of the Accushnet River".
Many of the ships destroyed were prizes captured by privateers operating out of the two towns. The conflagration caused by the fires they lit also destroyed homes and houses of worship, and was bright enough that it was seen in Newport, some 20 miles (32 km) away. During the evening the 38-man artillery garrison, manning a small fort on the Fairhaven side of the river (today known as Fort Phoenix), fired on the British ships, then spiked the fort's guns and abandoned it, leaving their colors flying. The British briefly returned fire, and then destroyed the fort's guns.
Grey's troops marched around the head of the Acushnet River to the eastern banks and camped. The next day they reembarked on their boats, but General Grey decided that Fairhaven should also be raided. In the meantime, militia were beginning to arrive to defend Fairhaven, with Major Israel Fearing taking command from an elderly colonel who was reluctant to pursue an active defense. When the British approached Fairhaven on the morning of September 6, Fearing arrayed about 150 men between the village and their landing point. After setting fire to a few nearby buildings the British headed for the village. At this point Fearing's men unleashed a withering volley of musket fire, and the British precipitately retreated to their boats.