Photogrammetry

Photogrammetry (sometimes called “photoscanning”) gives you significantly more accurate displacement maps than Bitmap approximation or Multi-angle techniques at the cost of being the most demanding way in terms of hardware, software and time. It works by taking pictures of an object from different angles and using them to create a representation of it’s geometry. In other words: It’s an incredibly accurate way to capture displacement data for material creation.

Introduction

The workflow shown in this guide works by creating a high-detail pass with the maximum amount of information possible and a smooth/flat pass. The details of the high-detail pass are then baked onto the low-detail pass, thus eliminating larger height changes and leaving only the smaller detail which creates uniform and easily tileable displacement maps at very high resolutions (theoretically up to ~32000px).

Deciding whether photogrammetry is the right tool

Photogrammetry can generally be used on surfaces which fulfill these criteria:

• The surface must have a strong enough displacement for it to create a detectable difference in perspective when moving the camera. What this means in concrete terms depends on your equipment.
  • Paving stones, bricks and tree bark are pretty much always suitable, even with lower end cameras.
  • Gravel and tiles with shallow seams are more difficult, but still doable.
  • With a high end camera, a macro lens and a lot of patience it is even possible to use photogrammetry for leaves and fabrics.
  • Plain/smooth wood is where photogrammetry hits its limits (at least for me right now) as the displacement changes are so subtle that they get lost in the noise during processing.
• The surface must not change it’s appearance (color/structure) based on the angle from which it is looked at. This rules out reflective and partially transparent surfaces.
• It must provide enough detail for the software to track camera movement from one shot to the next. Smooth paint or plaster can be a challenge.

Requirements

Hardware:

• A decent camera (DSLR or Mirrorless, I’ve never tried it with a smartphone)
• A tripod or monopod (I personally prefer the monopod for most situations)

Software:

• Metashape (or comparable photogrammetry tool)
• xNormal (or comparable baking tool)
• Affinity Photo (or comparable image editor)

The shooting process

Shooting for photogrammetry in Metashape works similarly to shooting for bitmap approximation.

Find a suitable surface and record a serpentine pattern in which every image has a decent overlap with the next one. One big point to remember during all of this is that Metashape needs differences in perspective to properly reassemble a surface. The Agisoft Metashape User Manual specifically points out that the different images should not be created by just rotating the camera around one point. It must be moved in 3D space for every shot to achieve perspective changes.

This also leads to the reason that I prefer using a monopod over a tripod for photogrammetry: 2D image stitching generally suffers from perspective distortions that appear if the camera is not always facing down at an exact 90° angle. Metashape is actually embracing these small differences! A lot of the time the small perspective changes introduced by using a monopod which isn’t always facing down in a perfectly straight angle help the software to reconstruct the surface.

Photogrammetry processing in Metashape

Once you have recorded your images the photogrammetry process can begin. The workflow I am describing here is how I do it in Agisoft Metashape but it should also be possible to perform similar steps in another photogrammetry tool.

We will begin with the detailed pass and then decimate and smooth it to get the flat pass for baking. Metashape batchjob files for this process are available on my Github but I would suggest performing the steps manually at least once as it will help you when doing troubleshooting in the future.

Metashape settings

Before doing any kind of processing in Metashape I would recommend adjusting a few settings. Go to Tools → Preferences and make sure that some settings are configured properly:

• The GPU should be enabled
• The Default view (in the “General” tab) should be changed to “Dense Cloud” as the default setting (“Model”) can cause Metashape to be extremely laggy when opening a large project.

Creating the detailed pass

Aligning photos

Import the photos and align the cameras via Workflow → Align Photos. The most important setting here is the quality preset. I would recommend using the “high” preset since all other presets either down- or upscale the given images. In the Advanced menu there two more options labeled Key point limit and Tie point limit. The Key Point limit defines how many points the software will extract from every photo and the Tie Point Limit then tells it how many of these points to use for the actual reconstruction. At this point you might be tempted to increase these settings to really high values but according to the Agisoft Metashape User Manual (page 22) these settings are primarily designed to tweak the performance rather than the quality, increasing them will therefore only have a marginal impact on the result (unless the reconstruction fails completely).

A “Sparse Cloud” in Metashape.

Dense cloud creation

Once the alignment has finished take a look at the result. In the best case you now have a rough representation of the surface made of a couple hundred thousand points. Choose the next item in the Workflow menu: Build Dense Cloud. In this pop-up select the quality of the dense cloud you want to create. This has a huge impact on the final result so I like to go for the Ultra high setting. This is also one of the more taxing parts for the computer, so save the project and then just play around with it until you find something that finishes within a reasonable amount of time (and by “reasonable” I mean less than 5-15 hours, it’s normal for it to take REALLY long.) Once it has finished you are getting your first real look at the scene. These are still just points but there are now so many of them that it is starting to look like a real model. All we need to to is connect them up to build the mesh.

A “Dense Cloud” in Metashape.

Building the mesh

From the “Workflow” menu select - you guessed it - “Build Mesh”. Here are some settings which need to be adjusted: Start by setting the reconstruction method to “Height field”. This means that there will only ever be one vertex on the Z axis for every pair of X and Y Coordinates, which is not only faster by a big margin but it’s actually something we are going to need anyways since our end goal is having a height map an not a 3D model. When it comes to choosing the quality I always go for the “High” preset but this is once again something you have to try out with your computer - and your hard drive - as the resulting files can go into the double digit gigabytes, so make sure you have some disk space left.

An untextured mesh in Metashape.

Building the texture

It’s time to take another step downwards in the workflow tab and click on “Build texture”. I usually generate at a resolution of 24576px or 32768px, this resolution allows me to make large crops in my images and still maintain a resolution of 8192px for the final texture.

A textured mesh in Metashape.

Exporting

For once, exporting does not happen in the “Workflow” menu but via “File” → “Export”. In this dialog box you can export both the texture and the model file at the same time. Save the model and the texture and name them “_DETAILED”. Once the first pass has been built you can start working on the second (flat pass).

Creating the flat pass

Decimating and smoothing the model

The second pass needs much less detail so the first step is to decimate the mesh. In Metashape this can be done via Tools → Mesh → Decimate. For the flat pass I always reduce it down to 200000 polygons. After that you need to smooth it. This is the key step that makes the baking possible. I always go for a smoothing strength of 2500.

Rebuilding the UV

Decimating the mesh unfortunately removes the UV Map from the model and the only way to rebuild it in Methashape is by actually creating a texture. The resolution of the second texture is pretty much irrelevant, 256px is just enough. This time it is important to select “Orthophoto” as the projection method as this will create one continuous UV map for baking.

Exporting

Save the flat pass as “_FLAT”. You don’t need to include the texture in this export.

Baking in xNormal

Now that you have both passes you can bake the detailed model onto the flat model. For that purpose I would recommend xNormal for its ability to handle very large meshes due to it lacking a 3D viewport which couldn’t handle the multi-gigabyte exports from Metashape. And it’s not like we need it anyways, because we know that the models are nicely aligned.

Open up xNormal. On the “High-Poly Meshes” screen right-click anywhere into the black field and select “Add High Poly Mesh”, then load your High-Poly OBJ file. It will appear in the list. Then right click on the file in the list (It’s important that you click on it and not below it) and select “Base texture to bake”. Load the detailed texture. It should appear together with the mesh in one line.

Then move on to the “Low definition meshes” section and load the flat pass in there.

After that you can choose the appropriate baking settings in the “Baking options”. Choose resolution, export path and a file format with a high bit-depth like EXR (Make sure your image editor of choice can actually open EXR files!) or PNG. Select “height map” and “Bake base texture” as the maps to render (You can of course add more maps but I try to keep the amount of different maps down at this point and generate all the other maps once color and displacement are made seamless).

asked by a pop-up window to adjust the levels for the height using two sliders. Change the minimum and maximum until all the parts of the height map that you want to use are neither completely black nor completely white (If they are it’s called “clamping” and it means that the area will appear completely flat in the export). You can enable the checkbox “Debug min/max clamping” to get a visual indication of which areas are currently clipping. It’s fine if the edges of the texture are clipping (those will be cropped away anyways), just make sure that the central area is nicely balanced. The image below shows the “Tonemapper” pop-up with the “Debug min/max clamping” checkbox enabled. The blue areas indicate where clamping is happening.

Making it seamless

Tiling both a color- and a displacement map at the same time is more difficult as you need to make sure that the two stay “in sync” during the entire process. For this reason I would recommend using an image editor with support for macros (like Affinity Photo or Photoshop). Open the raw color and displacement map as two layers. The first step should be to mark out the rough area that you want to use using guide lines. A bit like this:

Rough area of the material. The guiding lines have been exaggerated to make them easier to see.

There should be a bit of free space as some part from outside the selected area will be moved into the texture.

If your texture has a repeating pattern you should make sure that it is aligned to the grid. Here is an example for this adjustment: The bricks have to be aligned to the guide lines. Make sure to start recording your steps as a macro and perform a perspective correction to align the pattern to the guiding lines.

These bricks are not perfectly aligned to the guides. This will cause issues when trying to tile the surface

Fixed: The bricks are now aligned to the guiding lines.

Apply the same perspective correction to the other layer (that’s what the macro is for). Delete the macro that you just recorded. You can restart it, but it’s not required for the next part.

Select an area right above the target area (orange) and copy it into a new layer and move it down into the bottom part of the target area (green), like shown in this picture.

Then mask away parts of the orange area to hide the seam:

Repeat (or in the case of a using a macro replay) these actions for the displacement layer: Copy the selection from just above the target area down to the bottom edge of the target area, mask pars of it away using the same mask as the color map.

Then merge the both the two color layers into one and the do the same for the two displacement layers so that you are back to having only one color layer and one displacement layer.

For the horizontal tiling repeat the process on both the color and displacement layer by copying the the area right next to the box to the left part inside the box.

At this point you have a seamless color and displacement map inside the area your marked out. You can now crop your image down to the target area and export the two layers as separate images.

Creating the other maps

Does Ambient Occlusion make sense?

Having an ambient occlusion map is only useful if your surface has strong displacement that causes crevasses to be visibly darker by receiving less ambient light. If that’s not the case (for example in the case of an almost entirely flat parquet) then you can disable the ambient occlusion map.

Delighting the color map

Delighting is not always necessary for photoscanned textures since many photogrammetry tools already have built-in delighters. However, adding the delighting can still help if the photogrammetry software’s delighter didn’t catch everything. It can work in many different ways - Substance Alchemist even offers an AI based delighter. But the two most common methods are based on the displacement and direction of light and/or the ambient occlusion. In Substance B2M they are called “Light Equalizer” and “AO Cancellation”.

The AO Cancellation uses the ambient occlusion map to determine which parts of the image would naturally be darker in the unprocessed texture and then brightens these parts. I primarily use this method when creating textures.

Creating a roughness map

There are two ways to get roughness data. Roughness can be approximated from the displacement map (or rather the curvature which is derived from the displacement map). This method is great for creating small differences on surfaces with fairly uniform roughness. In Substance B2M this method is referred to as “Roughness from Curvature”.

It can also be influenced by the color of the texture. One thing I like to do when working with bricks (for example) is to use random color variations between the bricks to create randomness in the roughness map which would otherwise be very plain due to the bricks uniform structure. Here is what that can look like (Bricks 016):