Crews were called to the factory in Brick Makers Drive, Wollert just before 6.00am.
Station Officer Ken Houston said the fire involved a large brick conveyor, which went through multiple levels in the factory about 40m x 30m and 15m high.
He said there has been significant damage to the factory.
The fire was called under control at 7.00am and crews remained on scene for some time blacking out.
Five trucks from Craigieburn, Epping, South Morang, Greenvale and Wollert attended, as well as support from Sunbury and MFB crews.
The cause of the fire is unknown at this stage.