The Facts about Generative Fill AI in Photoshop

[Chris Sawyer]

Following on from our meeting on Tuesday...Barrie received the following video about AI in Photoshop.

Hi Barrie,

Admittedly not my usual kind of email, but with the recent release of Photoshop 2024 and with it (surprisingly) the move from Beta to include both Generative Fill and Generative Expand, I’ve put together one video already explaining why I think this has happened way too soon and all seems a little bit rushed.

The video is just over 8 minutes long and in it I cover the facts about where we are with Generative Fill as of today with regards to quality of results, and also Adobe’s plans for the future, PLUS show the results from a couple of suggestions for better results.

Glyn Dewis


https://www.youtube.com/watch?v=c5bbigpwIj4

[Bazza]

Extract from latest Amateur Photographer magazine - no photo images copied

++++++

IN THIS ISSUE / TECHNIQUE USING AI TO EDIT IMAGES
AI editing and how it’s changing photography
Will AI help photographers or replace them? Rod Lawton looks at how AI is being used in photo editors
f014-01
BEFORE

AI in photography is certainly a hot topic. There seems to be all sorts of different responses to this rapidly emerging technology, from fear and mistrust to excitement and enthusiasm.

In this feature we will take a look at the ways AI technology is currently being used in photographic image editing so that you can decide just how far you want to take it.

Make things better
AI can offer technical enhancements to images that were not possible before. AI can convincingly remove high ISO image noise while improving details, upsize images and improve the detail rendition in raw images over the original file. This is all achieved with AI deep learning software which can, essentially, work out what’s in the image and add to it, replace it or enhance it ‘intelligently’. We look at some of these tools in this first section.

Enhance and exaggerate
In the second section we’ll look at AI time-savers, such as AI masking tools, Lightroom’s Adaptive Presets and AI sky replacement. Photographers have long used editing software to change and enhance parts of a photograph, and AI tools now make this a whole lot simpler.

Change the world
This is where the AI controversy starts to bite. There are AI image generators like DALL-E which can create entire images from text prompts, but for photographers Adobe’s new Generative Fill and Generative Expand tools are probably more relevant, as ways to add new objects, backgrounds or surroundings to photographs that already exist. In the last section we’ll look at how these work, and what they might mean.

f015-01
AFTER

AI noise reduction has probably had the biggest impact on image enhancement, transforming this unusable ISO 6400 image from a Nikon D5600 into a smooth, crisp, high-quality image

f015-02
What is a Linear DNG?
Photographers want the best of both worlds. We don’t just want to enhance our images, we also want to keep our raw files with all their extended colour depth and dynamic range. Now even though you can process raw files in a multitude of ways, you still end up with a processed image which is no longer raw. Or do you? DxO and Adobe both use a special ‘Linear DNG’ file format which is an edited version of your original raw file, but still a raw format with all the advantages of raw. DxO PureRAW uses this format for its combined image correction and denoising, while Adobe uses it for Lightroom’s Denoise, Raw Details and Super Resolution features. This is a noise-free ISO 6400 Linear DNG created in DxO PureRAW and opened for editing in Adobe Lightroom Classic, where it behaves exactly like a regular raw file when editing.

USING AI TO ENHANCE IMAGE QUALITY
f014-02
DeepPRIME XD in DxO
DxO is not the only company offering AI noise reduction, but its DeepPRIME XD process is the best we’ve seen. It can be applied within the DxO PhotoLab 6 Elite photo editor, or via DxO’s PureRAW 3 batch processor. We used it on this ISO 12,800 image and it comes out looking noiseless and sharp.

f014-03
AI Denoise in Lightroom
Like DxO’s software, Lightroom’s Enhance feature works on raw image files and produces linear DNG output files which still behave like raws. This image was shot on a Nikon D5600 at ISO 6400 and is too noisy to use, but the enhanced version reveals an amazing transformation.

f014-04
Lightroom’s Enhance panel
Lightroom’s Enhance panel contains two further options (you can only click on one at a time). The Raw Details option was the first and definitely looks the weakest. On some image files you can see a slight difference, but on this one it’s so small that it hardly seems worth the effort.

f015-03
Enhancement tools
A side-by-side comparison reveals that there’s no real gain in our ‘enhanced’ version (left). You could easily improve on this with regular sharpening tools. Other software vendors have their own AI enhancement tools, and Topaz Sharpen AI is one example. It’s probably best not to expect miracles, though.

f015-04
Super Resolution in Lightroom
The Super Resolution tool is designed to upscale images and not just attempt to preserve edge definition (the old way) but to add image detail that was not recorded in the first place, using AI subject recognition to work out how missing detail would look.

f015-05
Detail enhancement with Adobe Enhance
Zooming in on the enlarged image versus the original shows a remarkable increase in actual definition and detail. Topaz Gigapixel AI and ON1 Resize AI are designed to do the same job. Image upscaling has really benefited from the AI revolution.

AI in software: Enhance and exaggerate
f016-01
AFTER

Sky replacement was one of the first uses of AI, and it’s a great example of how AI masking tools can dramatically speed up the editing process

f017-01
BEFORE

One of the first and best-known uses of AI in software was the AI Sky Replacement feature in Skylum Luminar. Photographers have been replacing bland skies in landscapes for decades, but traditionally it’s taken a good deal of time and skill, so most of us probably didn’t do it that often – or not at all if you’re the sort of photographer who objects on principle.

Sky replacement is a technique that also involves replacing objects in a scene to alter ‘reality’, though in this instance it simply layers in other photographs. It’s a compositing technique rather than AI content generation.

For this section, we’ll treat sky replacement as an early example of how AI has been used to change the way we mask images – or to be more precise, how we can get the software to do it for us.

On a basic level, this can save us a lot of time but, more than that, it can encourage us to try more advanced editing techniques and manipulations that we might not have attempted before.

The AI masking tools in the latest versions of Lightroom take this to new heights. Lightroom can’t do sky replacement, since this involves layering different images, but what it can do is automatically mask all manner of different subjects for individual editing and enhancement. It can automatically identify skies, backgrounds and what it thinks is the photograph’s subject (with remarkable success) and can even isolate and mask individual objects.

AI masking tools can change the way we edit photos – how long it takes and even what we attempt. It makes masking easy where before it was difficult, so some expert editors may object to their ‘craft’ being eroded, but that doesn’t seem a good reason to reject this new AI assistance.

Lightroom’s Adaptive Presets
Adobe Lightroom comes with many one-click adjustment presets which apply a fixed set of global adjustments to the images you apply them to. But with the arrival of AI masking tools, Lightroom has a new trick – ‘Adaptive Presets’. These use a combination of automated AI masking and local adjustments to identify areas on your images that the adjustments should be applied to. Here’s an example. The ‘Storm Clouds’ preset in the ‘Adaptive: Sky’ section of our presets list has selected the sky only to increase the drama and contrast in this sky while leaving the rest of the photo unaltered. There are Adaptive Presets for automatically recognised ‘Subjects’ and ‘Portrait’ features from skin tones to black & white and even group portraits. You can make your own Adaptive Presets too.

f017-02
‘AI masking tools can change the way we edit photos - how long it takes and even what we attempt. It makes masking easy...’
SKY REPLACEMENT AND AI MASKING
f016-02
Sky replacement in PS
This is the sky replacement tool in Photoshop in action. You don’t need the Photoshop 2024 beta for this as it’s already part of Photoshop 2023. The results could look a little crude in the early days, but now they can look very good, especially if you choose a sky with similar colours to your subject.

f016-03
Mismatched skies
If you do choose a sky that doesn’t match your subject, the masking will be just as effective, but the image will still look wrong, and the Sky Replacement panel’s Foreground Adjustments don’t really go far enough in matching the foreground tones to the sky (Luminar seems a little better at this).

f016-04
Manual tweaks
All is not lost, though. Photoshop adds its sky replacements as a series of layers and masks, so it’s easy for us to add a Photo Filter adjustment above the base image layer to warm up the photo and a Curves adjustment at the top of the layer stack to increase the overall contrast.

f017-03
Lightroom Sky mask
Sky replacement is an early example of AI masking in action, but now Adobe has introduced a whole suite of AI masking tools to Lightroom. For example, in this shot we can add a Sky mask to automatically identify and select the sky in our image to reduce the exposure and stop it clipping to white.

f017-04
Lightroom Subject mask
Lightroom can also automatically identify and mask what it thinks is the main subject in a photo, and it gets it right almost every time. Here, we created a new Subject mask and Lightroom figures out this tower must be the subject and creates a mask without any further work needed from us.

f017-05
Lightroom Object masks
Here’s another AI masking type – Object masks. Here, you use a brush tool to very roughly paint over the area occupied by the object you want to mask and the AI then identifies and precisely masks that object. In this case it’s the trees in the background, which we’ve made lighter and more vibrant.

AI in software: Change the world
f018-01
AFTER

Photoshop’s new Generative AI tools have been used here to extend a simple seascape upwards and sideways and to add a sailing ship, forest and birds

These days sky replacement tools are common. Skylum Luminar has led the field, and ON1 now has Sky Swap AI built into ON1 Photo RAW and also as a Lightroom plug-in. Photoshop has its own in-built sky replacement process too.

However, sky replacement was just the start. Now it’s possible to add objects to a scene and extend backgrounds with Adobe’s still-experimental ‘Firefly’ generative AI, and the Generative Fill and Generative Expand features that are now part of Photoshop 2024.

Adobe is not trying to create pseudo-photographic images from scratch in the same way as an AI image generator like DALL-E, though it does use the same principle of text prompts to generate new objects and image regions, coining the term ‘promptography’.

f019-01
BEFORE

With the current state of the technology, this can either work extraordinarily well or not very well at all. As with many of Adobe’s ‘content-aware’ tools in the past, it depends on the area selected, the surroundings and what the algorithms make of them. It works well or it doesn’t, and if it doesn’t work well then you can try a slightly different selection or area and try again.

Adobe’s AI is currently very good at generating ‘generic’ objects, but is very wary of brands and object designs associated with brands, and with renderings of people and faces – which can often look like something out of a horror flick, so be warned.

The Generative Expand feature, however, is quite remarkable, and could change the way that you crop around subjects and work with multiple aspect ratios for print or web.

Is this cheating? Is this reality? These are philosophical questions that we all have to think about. Let’s not kid ourselves – it has always been easy to cheat in photography and distort or fabricate reality. What has changed is that now everybody can do it, and they can do it more convincingly than ever before.

Photoshop 2024
Photoshop 2024 is now out of beta and you can download it via the Adobe Creative Cloud app. We weren’t prompted to update our existing version of Photoshop, though. Instead, we found PS 2024 in the Firefly and Generative AI tab, where it downloaded alongside our 2023 version instead of replacing it. The Generative Expand and Generative Fill do seem to work rather better than they did in the Beta version where Adobe’s AI tech was first released. There were fewer bad joins in this finished version, though it could still take a few attempts to find a satisfactory expand or fill result. Usefully, Photoshop adds its new generated AI imagery on new layers over the original image, together with a mask to blend it in – and the new layers are named after the prompt you used to generate them, which is a nice touch.

f019-02
GENERATIVE EXPAND AND FILL IN PHOTOSHOP
f018-02
Re-cropping with Generative Expand
Here’s a simple example of how the Photoshop beta’s new Generative Expand can be useful. This picture was shot in landscape format, but what if we need a square image for Instagram? We can create a new crop that extends up into the sky, and Photoshop will use AI to fill it. We can prompt it with a word or phrase, or leave it to decide for itself.

f018-03
Prompting Generative Expand
Photoshop has done a great job of filling in our image with more ‘sky’, but let’s take it a bit further. This time we’ll extend the photo sideways to make a 16:9 ratio image, and prompt Photoshop with the phrase ‘sailing ship’. Now it will use AI to fill the extra ‘expanded’ area, but try to include an AI-generated version of the object we asked for.

f018-04
Generative Fill in Photoshop: choose a result
You don’t have to accept Photoshop’s first attempt. It will offer three alternatives in the Generative Layer panel on the right; you can click the Generate button again to generate three more, and so on until you’re happy. Our ship looks fairly natural and has been blended in well, though it took a few clicks of the Generate button to find a good outcome.

f019-03
Generative Fill with prompts
That’s the Generative Expand feature in action, but you can also add or replace objects in existing areas. Let’s see if we can add a forest to the bare headland in the distance. We start by creating a rough freehand selection around the hillside, then type the phrase ‘forest’ into the floating ‘Generate’ panel.

f019-04
Generative Fill results
What Photoshop has done here is pretty remarkable. It has inserted a hillside full of natural-looking conifers and blended them perfectly with their surroundings. The scale of the trees, their colours and tones, the lighting – everything is exactly right. It’s hard not to be impressed by what Adobe’s AI Generative Fill has achieved here.

f019-05
Birds and other objects
We’ll do one more thing, selecting an area of sky and then using the prompt ‘birds far away’’ to fill it. Sometimes it takes a couple of attempts to find the right form of words for the AI to respond to, but the result is pretty good. But what kind of birds are they? Who knows? Photoshop’s generative AI creates convincing but ‘generic’ objects. ■

[Chris Sawyer]

Thanks Barrie...long article.   Some useful LR/PS tips which I definitely will not be using for competitions.

 I had to imagine what the example photos looked like using MI (My Intelligence).

[Fay]

should have added the photos Barrie

[Bazza]

Yep agree, will try another method but its from digital version of magazine