Adding details on a large world that works in VR

Recently I added a new spy mission to Dogfight Elite. The idea was simple: allow players to unmount their plane and find a spy that is hiding inside an abandoned subway tunnel. The problem is that the tunnel is quite large with tons of assets and custom lighting. At the same time, outside the tunnel, you have a large terrain and multiplayer combat going on. This must run at 60 fps on Oculus VR.

How did I solve this?

There were several steps:

1. The usual draw calls, shader, and mesh optimizations. Most items inside the tunnel share textures and materials. They are highly optimized for polygon counts. The items also utilize the same shaders with the exception of the lights. Everything that should be static, is marked as static. Batching was easy to achieve.

But this was not enough. I had to write a custom dynamic loading/unloading system for the assets in memory. The RAM was not enough to load all the assets and Unity culling is not smart enough to avoid rendering distant objects.

2. Based on distance and orientation, load and unload items. Also, enable/disable assets that will require no rendering. I had to create an in-memory grid coupled with some triggers. This way I could achieve things like: when you enter this area, you can stop rendering the water, terrain, and trees. You will not see them when deep inside the subway, but you need to see them when you look out from the large subway entrance.

You can see here a video while developing the dynamic system. When the pilot arrives and enters, it begins to dynamically load the assets while walking inside the tunnel. The assets also unload from memory when you leave those areas. In the next iteration, I also stopped rendering water, trees, and so on, as you go deeper into the tunnel. But I could not unload them from memory for the reasons that I will explain below.

So far it all seemed good. Framerate was good. RAM was good. I could run this even a Kindle Fire 7 at a decent speed. But then I arrived at the hardest problem to solve: hiccups.

In Unity, when you load an asset into memory for the first time there is always a hiccup. It does not matter if you load them asynchronously. Unity has a design flaw that when you add a new GameObject, it instantiates/runs initialization code in the main thread. There is no way around this. The Monobehaviour class is just badly designed for this scenario. You can hide it by preloading or “warming up” different assets and then unloading them. But the problem with this is that you will need a large amount of RAM and make people wait a long time on their loading scenes. In my case, I also have a terrain with trees. When I unloaded the terrain from memory and I loaded it back upon exiting the tunnel, the hiccup was huge. So how can we solve this?

Well, it is certainly a known problem. After years of complaints, Unity decided to address it by creating a new system called DOTS/ECS, or Data-Oriented Technology Stack and Entity Component System. This was what seemed the right approach. Unity developed several demos of the technology, including this one loading/unloading thousands of assets in real-time with no hiccups.

But then, it lost traction and Unity stopped talking about it or releasing updates on its progress. Until recently. You can follow the thread in the forum here:

https://forum.unity.com/threads/dots-development-status-and-next-milestones-march-2022.1253355/

With this announcement, Unity acknowledged this is a standing problem in their design and they will include DOTS as part of Unity’s core. This should be the way to go for large world asset loading now. Development is still in the early stages but you can already use it in your projects. You can follow the roadmap and download the latest packages in:

https://unity.com/roadmap/unity-platform/dots

Advertisement

Secret Photo Folder App fix coming

First things first, if you are one of the thousands of current users coming here to check for the status, I’m pleased to announce that it is fixed and under review by Apple. It should be published in the next 48 hours.

The app should be available as an update in the store. Direct link here: https://apps.apple.com/us/app/secret-photo-folder/id367327563

• 

Now, let me explain the issue.

Apple recently released iOS 15.5 and I had to update many of my apps. If you are familiar with Apple App development, you know that Apple enforces developers to constantly update the apps in the store otherwise they get removed. This is the reason why even though I’ve developed more than 80 apps, you will find only a few in the store. I don’t have time to keep updating all of them and Apple removes them as time passes by.

This system has some advantages, mainly that Apple makes sure your apps are always updated to the current iOS. But it also comes with two serious caveats:

1. Most people do not have the time and resources to keep updating apps that do not really need any updating if not enforced by Apple.
2. Apple constantly deprecates and breaks features with new iOS updates. Even if your app hasn’t changed a bit, it will begin malfunctioning unless rewritten with the newer iOS. This brings me back to point 1 and you enter the endless loop.

Enter my app: Secret Photo Folder.

I developed this app years ago. The app allows you to import photos and videos, organize them into folders, add descriptions and keep them hidden with a password and outside your camera roll. It has millions of downloads and it worked perfectly. No issues whatsoever with the app. Well, no issues until Apple released the iOS 15 update. With the iOS update, I begin receiving emails that the “forgot password” option is not working any longer. Apple decided that when you press “face id” to identify yourself, you have to add a text saying something as silly as “We require Face Id to identify you”. As if pressing “face id” was not clear enough of what was about to happen.

So I load my Xcode, modify the app to add that line, compile and… errors. Tons of them. Apple has deprecated many of its APIs that handle photos and videos. And my app is mainly a photos and videos app! I find myself in a bad place: my app is working fine, but if I don’t update it to the latest iOS, it will be removed soon. On the other hand, to update it, I need to spend weeks developing and rewriting thousands of lines of code to use their new APIs. Doing the latter means I need to retest everything almost as if it was the first time I wrote this app.

After some thought, knowing that this app is used by many people I decided to bite the bullet. Weeks later I have the update ready and submit it to Apple. And guess what… that same day the app is updated, iOS 15.5 update hits the public and they included several more bugs. In particular, the app is not even able to import photos!

I’ve spent the last 3 days upgrading again the code to the latest iOS. The update is ready and submitted but the users (thousands of them) are not happy (rightly so) with the current version. I’ve gotten over 1000 emails and reviews in the app are horrible. I hope Apple releases the update asap. The problem is that Apple keeps doing this release after release. If you check the app reviews throughout the years, you will see reviews complaining about “photos can’t be imported anymore” or “colors changed”. In 2017, 2019, and now 2022. Ironically, nothing changed in the app. It was Apple making changes that broke the app with its iOS updates. Most developers give up and their apps get delisted. Myself included, I’ve let over 50 apps get delisted at this point because I don’t have enough hands to keep up. I will continue updating Secret Photo Folder because it is used by many, but oh boy! it gets hard!

If you are a developer and are having a similar experience, let me try to help you out a bit.

This code now fails to work in iOS 15.5:

[PHAsset fetchAssetsWithLocalIdentifiers:@[assetId] options:nil];

You will probably get some errors when importing Photos or Videos. Instead you need to use the url with:

[PHAsset fetchAssetsWithALAssetURLs:@[_assetURL] options:nil];

This is crazy because the last line of code is now deprecated. In theory, the correct line is the first one. I will expect Apple to fix it in the next iOS update, but for now we have no choice.

Another issue, is that the photos and videos showed up rotated. The reason is because this line does not create images as they used to:

image = [UIImage imageWithCIImage:image];

Now you need to specify the scale and orientation. Otherwise you will find most of your photos loading up in landscape. The correct method now:

image = [UIImage imageWithCIImage:image scale:1 orientation:orientation ];

Lastly, videos used to be able to load and import them with a simple call to:

PHImageManager requestImageDataForAsset:self.asset options:options resultHandler:^(NSData *imageData, NSString *uti, UIImageOrientation orientation, NSDictionary *info)

This will only work for images with the iOS update. To import videos you need to handle them independently with:

PHImageManager requestAVAssetForVideo:self.asset options:nil resultHandler:^(AVAsset *asset, AVAudioMix *audioMix, NSDictionary *info) {
 
AVURLAsset* urlAsset = (AVURLAsset*)asset;
                                NSNumber *size;
                                [urlAsset.URL getResourceValue:&size forKey:NSURLFileSizeKey error:nil];
                                NSData *imageData = [NSData dataWithContentsOfURL:urlAsset.URL];

                   

And then, save that imageData to the destination folder. Happy development!