Jan 08, 2010

Replace Amazon S3 with MongoDB GridFS and Grails

Mostly because this blog started on a hosting plan that didn't really allow file server access from web applications, I got in the habit of putting files on Amazon S3. While I love the ease of throwing something on S3, I don't particularily like the cost given the small amount of data I'm serving per month.

My recent foray into MongoDB got me thinking about using it to serve files. You can do just that with MongoDB's GridFS specification. When a file is stored in GridFS, it is represented by a metadata object and one or many chunks which store a subset of the data. Chunking the files helps with searching but could help replication and sharding presumably. Files in a GridFS store are just like any object in the database. You filter by any of the metadata properties and get chunks on demand or out of order.

Setting up your application

You could hardcode config settings if you run mongo on a different host or port but I put the following settings in Config.groovy:

mongodb {
    host="localhost"
    port=27017
    dbName = "whatever"
}

I added the following to my UrlMappings.groovy file so expose a download path to our files.

"/downloadFile/$filename" {
    controller = "files"
    action = "downloadFile"
}

Wiring your application to MongoDB

Next I created a GridFSService to interface with MongoDB. To get the settings from Config.groovy injected, I subclassed InitializingBean and did my configuration in afterPropertiesSet. Keeping things simple, I'm just allowing saving a file, deleting a file, and getting a files list.

void afterPropertiesSet() {
    this.mongoSettings = grailsApplication.config.mongodb
    mongo = new Mongo(mongoSettings.host.toString(), mongoSettings.port.intValue())
    if (mongoSettings.bucket == null) {
        gridfs = new GridFS(mongo.getDB(mongoSettings.dbName.toString()))
    } else {
        gridfs = new GridFS(mongo.getDB(mongoSettings.dbName.toString()), mongoSettings.bucket.toString())
    }
}
boolean saveFile(file) {
    def inputStream = file.getInputStream()
    def contentType = file.getContentType()
    def filename = file.getOriginalFilename()

    try {
        if (gridfs.findOne(filename) == null) {
            save(inputStream, contentType, filename)
        } else {
            println "Removing old file and uploading new file"
            gridfs.remove(filename)
            save(inputStream, contentType, filename)
        }
    } catch (Exception ex) {
       throw ex
    }
    return true
}

def save(inputStream, contentType, filename) {
    def inputFile = gridfs.createFile(inputStream)
    inputFile.setContentType(contentType)
    inputFile.setFilename(filename)
    inputFile.save()
}

def retrieveFile(String filename) {
    return gridfs.findOne(filename)
}

def deleteFile(String filename) {
    gridfs.remove(filename)
}

def getFilesList() {
    def cursor = gridfs.getFileList()
    cursor.toArray()
}

The last piece is to create the controller we referenced in UrlMappings.groovy. The FilesController contains actions for uploading, downloading, and deletion. For brevity's sake, I'm just showing the actions for downloading and uploading.

def downloadFile = {
    def filename = params.filename
    println filename
    def file = gridfsService.retrieveFile(filename)
    if (file != null) {
        response.outputStream << file.getInputStream()
        response.contentType = file.getContentType()
    } else render "File not found"
}

def upload = {
    def f = request.getFile('myFile')
    println f
    if (!f.empty) {
        if (gridfsService.saveFile(f)) {
            redirect(action:'uploadComplete')
        } else {

            flash.message = 'Error occured during upload, please try again.'
            redirect(action:'uploadFile')
        }
    } else {
        flash.message = 'An empty file cannot be uploaded.'
        redirect(action:'uploadFile')
    }
}

At the moment, the above code only supports a single bucket but you could easily alter the service and configuration to support more. GridFS support will be a part of an upcoming Grails MongoDB plugin.