In some web applications, you might need to let users upload their own images and photos. Maybe for a profile picture. Or to share a photo. Or to add an image to a blog post, or some other user-generated content.
When you upload your own images for use on your websites, you probably try to make sure you upload them at the right dimensions. And you probably try to optimise the image so it’s not bigger than necessary.
Because massive image files are slow to load and cost more in bandwidth and storage.
But you don’t have that same control when users upload images. They might not even know how to resize or alter an image file. They might just grab a photo off their phone and upload it.
I had this same issue. To protect my server from massive images, I originally put some restrictions on upload size and dimensions. On the client, to tell the user if the image was too big. And on the server, to validate the image before I stored it.
Processing images in the browser
In the browser, I would check the file size first. If the file size was over 5MB, I’d let the user know they need to upload a smaller image. If it was under the file size limit, I’d secretly load the image and check the width and height. I limited both width and height to a 5000px maximum, and again if the image was too big, I’d let the user know it needed to be smaller.
let filesize = ((file.size/1024)/1024).toFixed(4); // mb
if ((file.type === "image/jpeg" || file.type === "image/png") && filesize < 5) {
// check the dimensions
let _URL = window.URL || window.webkitURL;
let img = new Image(); /* global Image */
img.src = _URL.createObjectURL(file);
img.onload = function () {
if (this.width > 5000 || this.height > 5000) {
window.alert("Your image needs to be less than 5000 pixels in height and width.");
} else {
handleUpload(file); // handle the upload
}
};
} else {
window.alert("Your image needs to be an image in jpg or png format and less than 5MB in size.");
}Then I found there were a few problems with this approach.
While it’s good to have some limits, the images are still allowed to be much bigger than I needed them to be. If you’re not displaying the images at 5000px, you don’t need to save them this big.
But limiting the size anymore at upload isn’t very fair on the user. They would have to figure out how to crop and reduce the size of their images before each upload. Some users might not be very technical or have the software to do this (especially if they are using a smartphone).
And to be honest, the 5MB limit was already getting a bit too restrictive. Mobile device cameras are getting better as the years go by, and the images they take are getting bigger.
I did consider processing images in the browser to reduce the size (using canvas), but in the end, I decided against it due to browser compatibility and memory concerns for users.
Processing images on a NodeJS server
The next, and only other option (other than using an external service), is to process the images on the server. After a bit of research, I found sharp. It seemed to tick all the boxes. Here’s how you can use it:
sharp('input.jpg')
.resize({ width: 300 })
.toFile('output.jpg');The resized image output.jpg would be 300 pixels wide, with the height auto-scaled.
I already have an express route that handles receiving the image file from the browser. This is how the code looks, using multer to validate and get the file, and multer-s3 to upload the image.
var aws = require('aws-sdk')
var express = require('express')
var multer = require('multer')
var multerS3 = require('multer-s3')
var app = express()
var s3 = new aws.S3({ /* ... */ })
var upload = multer({
limits: { fileSize: 5 * 1000 * 1000 }, // 5MB max file size
fileFilter: function(req, file, callback) {
let fileExtension = (file.originalname.split('.')[file.originalname.split('.').length-1]).toLowerCase(); // convert extension to lower case
if (["png", "jpg", "jpeg"].indexOf(fileExtension) === -1) {
return callback('Wrong file type', false);
}
file.extension = fileExtension.replace(/jpeg/i, 'jpg'); // all jpeg images to end .jpg
callback(null, true);
},
storage: multerS3({
s3: s3,
bucket: 'some-bucket',
acl: 'public-read',
key: function (req, file, cb) {
cb(null, `uploads/${req.file.originalname.replace(/\W|jpeg|jpg|png/g, '') : "" }.${file.extension}`); // removes non-word characters from filename
}
})
})
app.post('/upload', upload.single('file'), function(req, res, next) {
res.send(req.file.location);
});I figured slipping sharp in between the two would be the best way, but since multer sends a stream directly to multer-s3, I wasn’t able to get that to work. I think the reason is that sharp needs the file stored somewhere to process it.
You could get the file back from object storage, transform it, and then re-upload it. But I decided to save on bandwidth and avoid sending the large image out to be stored externally. You only need the file temporarily so you can either store it on the local file system or in memory. Since these original images might be large, I didn’t want to go for memory, so I chose to save a file temporarily on my server.
To do this, I removed multer-s3 so I could use local storage instead. Then I use aws-sdk to upload the image to s3 storage after it's been reduced in size.
Here’s the process:
- Get the original image (multer)
- Store the original image (
/tmp) - Reduce the image size (sharp)
- Save the processed image (aws-sdk)
Here are the packages you need, assuming you already have your server and routes up and running.
npm install --save multer aws-sdk sharp
And here’s the code:
var aws = require('aws-sdk');
var express = require('express');
var multer = require('multer');
var sharp = require('sharp');
var app = express();
var s3 = new aws.S3({ /* ... */ });
var upload = multer({
limits: { fileSize: 10 * 1000 * 1000 }, // now allowing user uploads up to 10MB
fileFilter: function(req, file, callback) {
let fileExtension = (file.originalname.split('.')[file.originalname.split('.').length-1]).toLowerCase(); // convert extension to lower case
if (["png", "jpg", "jpeg"].indexOf(fileExtension) === -1) {
return callback('Wrong file type', false);
}
file.extension = fileExtension.replace(/jpeg/i, 'jpg'); // all jpeg images to end .jpg
callback(null, true);
},
storage: multer.diskStorage({
destination: '/tmp', // store in local filesystem
filename: function (req, file, cb) {
cb(null, `${req.user._id.toHexString()}-${Date.now().toString()}.${file.extension}`) // user id + date
}
})
});
app.post('/upload', upload.single('file'), function(req, res, next) {
const image = sharp(req.file.path); // path to the stored image
image.metadata() // get image metadata for size
.then(function(metadata) {
if (metadata.width > 1800) {
return image.resize({ width: 1800 }).toBuffer(); // resize if too big
} else {
return image.toBuffer();
}
})
.then(function(data) { // upload to s3 storage
fs.rmSync(req.file.path, { force: true }); // delete the tmp file as now have buffer
let upload = {
Key: `uploads/${req.file.originalname.replace(/\W|jpeg|jpg|png/g, '') : "" }.${file.extension}`, // removes non-word characters from filename
Body: data,
Bucket: 'some-bucket',
ACL: 'public-read',
ContentType: req.file.mimetype, // the image type
};
s3.upload(upload, function(err, response) {
if (err) {
return res.status(422).send("There was an error uploading an image to s3: " + err.message);
} else {
res.send(response.Location); // send the url to the stored file
}
});
})
.catch(function(err) {
return res.status(422).send("There was an error processing an image: " + err.message);
});
});As you can see, I reduce images down to a maximum width of 1800px, since that’s as big as they need to be for my use case. You can alter this to what you need. By only passing a width, the height will also be scaled down at the correct proportions.
You might want to also set a maximum height and/or scale down images based on height too.
Now users can upload bigger images (up to 10MB), but the files that end up getting stored and displayed are much smaller. Win, win!