Reimplement batching in YesSql, Azure Application Insights module - This week in Orchard (26/02/2021)
Refactoring the Content zone, reimplement batching in YesSql, hiding Setup recipes, adding more indexes to index tables, Azure Application Insights module and a lot more is waiting just for you in this post!
Orchard Core updates
Do not display Setup recipes in the admin UI
You can easily list all of the existing recipes of your site under Configuration -> Recipes. If you create a recipe, like myrecipe.recipe.json, and put it in a Recipes folder of one of your modules, that recipe will be listed on this page. Let's add a simple one and check out the content of that page.
You can see that our recipe with the display name My Recipe is on the list but wait! What happened with the other recipes? We have several other built-in recipes like the Blog, the Agency, and so on. Where are they? From now the logic in the Index method of the AdminController in the OrchardCore.Recipes module is slightly changed. If the recipe is a setup recipe (defined in the issetuprecipe of the JSON file) or has the hidden tag, the recipe will not appear in that list. You can see that our myrecipe has the false value for the issetuprecipe but the Blog recipe has the true value.
Admin Dashboard Widgets
We mentioned the Admin Dashboard widgets two times in This week in Orchard that allows you to add cards to the homepage of the dashboard, which is about to represent a piece of functionality of a given feature or module. You can find the first post here and the second one here. And now this feature has been merged to the dev branch of Orchard Core and there is also a new page in the Orchard Core Documentation about this module. It hasn't got too much information yet but the embedded recording could help you to see this feature in action. If you haven't heard about this feature, don't forget to check out the previous blog posts and the demo video!
Use DocumentId in indexes
This fix is also about using the correct combination of fields for each index that we use. For example, the index for the UserIndex table now contains every field of the table. So, from now you will find indexes for every index table.
Refactor ContentZone - Tabs, Card, Column containers
With more and more dynamic shapes coming into Orchard Core (all the tab/card grouping shapes, and also shapes like the ContentCard) it would be potentially useful to have a ShapesViewModel that could be used as a hard type abstract class to create specific targetted ViewModels, i.e. a TabViewModel.
The problem with the ShapeViewModel is it doesn't support a list of positioned Items, so right now we are using Shape because many of these tab/card/content card shapes require a list of positioned items as well.
So we was not intending that a ShapesViewModel was actually an IShape, just an abstract that could be used to build shapes with, and the mixin would still turn it into an IShape, by mixing in the ShapeViewModel which has all the required IShape properties.
The idea being you could then use the IShapeFactory to create a shape, then add items to it.
var shape = ShapeFactory.CreateAsync<TabViewModel>("Tab", m =>{m.TabId = tabId});{
foreach(var item in thetabs)
shape.Add(item);
}
The current usage builds 3 different shapes, even when displaying on the front end, and there are no groups. Here we do a quick check first, to see if there are any groups, and if there are none, just render the zone directly. No extra shapes.
Also
- Removes uses a dynamic, and moves to hard typed models.
- Uses ToLookup instead of GroupBy (better usage).
- Uses fewer dynamics for injected properties, i.e. DisplayAsync can be resolved directly to its interface instead of being dynamic (this needs evaluation, as we might do it for the other C# Shape Attributes, or not).
If you check out the modifications of the ZoneShapes class, you will find the implementation of the ContentZone shape and building the TabContainer shape by using the new GroupingsViewModel.
And by having the GroupingsViewModel ViewModel we can use that instead of a dynamic one as you can see in the TabContainer.cshtml file.
Demos
YesSql: Reimplement batching
When we do updates on the content items and when one deletes a document all the index tables corresponding to the document type get a DELETE query. And this DELETE query is actually a single DELETE query. If we see a benchmark we can realize that the batching isn't working. So, if you would like to delete multiple content items, the code is just about to send multiple DELETE requests instead of sending one.
If you run the query locally using SQL Server and you have 1000 indexes, it takes one second. So, you might not notice that it's slow. It's slow but it could look like that's because we have several indexes.
Extra Indexes: 1, Elapsed 00:00:00.0371806Extra Indexes: 100, Elapsed 00:00:00.1317780Extra Indexes: 200, Elapsed 00:00:00.2826214Extra Indexes: 500, Elapsed 00:00:00.6213360Extra Indexes: 1000, Elapsed 00:00:01.2628232
But when you have the SQL Server in Azure, the deletion of 1000 indexes could take 28 seconds. At this point, you will start noticing the performance just after 100 indexes. Apparently, we have around 20 indexes in Orchard Core but you can quickly arrive at 100.
Extra Indexes: 1, Elapsed 00:00:00.0838498Extra Indexes: 100, Elapsed 00:00:02.5644737Extra Indexes: 200, Elapsed 00:00:05.0814065Extra Indexes: 500, Elapsed 00:00:12.6558409Extra Indexes: 1000, Elapsed 00:00:28.6622291
In YesSql, when you update a document, it needs to update all the indexes, which means it will rebuild them. And an index can return multiple records like if I'm indexing the name of someone, I might want to index the first name, the last name, the middle name. These are three records, one per name. So, what it does is, it builds the three records in memory and it will send a query to delete any record that was associated with the document and then sends three inserts for the new records. So you have one for the delete and three for the inserts. But it sends a delete even for the indexes that didn't return anything because the fact that you didn't return anything might mean that there is nothing anymore associated. So you need to delete and send nothing. This way we get a delete per index. That's not optimal but that's how YesSql works and that's optimal for reads and not writes. To goal is to make reads faster than writes.
Still, we should not have to send one independent query to do the deletes per index. That's what this issue is about. And Sébastien Ros managed to fix it. And here are some numbers. These are all for dummy indexes that never perform any writes. Just cause deletes. What we can see in the profiler is that everything easy to batch is batched together. The big batch with mostly deletes, and a couple of inserts, is now long-running, instead of many many short runs. Probably from the look of it, the insert is still expensive, so which pushes up the time run.
Before batchingLocalIndexes 1, Elapsed 00:00:00.3004236Indexes 100, Elapsed 00:00:00.2375774Indexes 200, Elapsed 00:00:00.3583902Indexes 500, Elapsed 00:00:00.7695818Indexes 1000, Elapsed 00:00:01.2836934
Remote
Indexes 1, Elapsed 00:00:00.7207663
Indexes 100, Elapsed 00:00:03.3552247
Indexes 200, Elapsed 00:00:05.5547927
Indexes 500, Elapsed 00:00:13.8364514
Indexes 1000, Elapsed 00:00:27.2306443
After Batching
Local
Indexes 1, Elapsed 00:00:00.2207824
Indexes 100, Elapsed 00:00:00.0910920
Indexes 200, Elapsed 00:00:00.1632908
Indexes 500, Elapsed 00:00:00.4007200
Indexes 1000, Elapsed 00:00:00.4559752
Remote
Indexes 1, Elapsed 00:00:03.6326000
Indexes 100, Elapsed 00:00:04.9639312
Indexes 200, Elapsed 00:00:09.8273422
Indexes 500, Elapsed 00:00:16.1340951
Indexes 1000, Elapsed 00:00:15.2008296
And how it looks like is the following. If you create a blog post in Orchard Core and after the change, there is a single communication to the database that contains everything that you can see here. This contains creating the Document, creating the ContentItemIndex, updating the ContentItemIndex with the DocumentId, and so on. These are three indexes to update (ContentItemIndex, ContainedPartIndex and AutoroutePartIndex).
insert into [Document] ([Id], [Type], [Content], [Version]) values (19, 'OrchardCore.ContentManagement.ContentItem, OrchardCore.ContentManagement.Abstractions', '{"ContentItemId":"4cpw0fnmjb1kp07dmzxx8n8ecg","ContentItemVersionId":"4953p18bj3gyy5yy82f7mj7w4y","ContentType":"BlogPost","DisplayText":"The title","Latest":true,"Published":false,"ModifiedUtc":"2020-12-31T00:31:34.3346095Z","PublishedUtc":null,"CreatedUtc":"2020-12-31T00:31:34.3346095Z","Owner":"48v9vt5vxznr5z9m1df9zmvjm8","Author":"admin","TitlePart":{"Title":"The title"},"AutoroutePart":{"Path":"blog/the-title","SetHomepage":false,"Disabled":false,"RouteContainedItems":false,"Absolute":false},"BlogPost":{"Subtitle":{"Text":"Subtitle"},"Image":{"Anchors":[],"Paths":[],"MediaTexts":[]},"Tags":{"TagNames":["Space"],"TaxonomyContentItemId":"4ykev5wxfcny7tvsahz9y64mwe","TermContentItemIds":["4nv0z7r24r1vw3sfpq7t6xws59"]},"Category":{"TaxonomyContentItemId":"4tpy2wv97bkbf0zkx8tyd1bm4q","TermContentItemIds":["4bsstr09f29rp0sgy85n9f07wj"]}},"MarkdownBodyPart":{"Markdown":"Some text"},"ContainedPart":{"ListContentItemId":"491emynv0kavbzhy40xmqv1wds","Order":0}}', 1);insert into [ContentItemIndex] ([ContentItemId], [ContentItemVersionId], [Published], [Latest], [ContentType], [ModifiedUtc], [PublishedUtc], [CreatedUtc], [Owner], [Author], [DisplayText]) values ('4cpw0fnmjb1kp07dmzxx8n8ecg', '4953p18bj3gyy5yy82f7mj7w4y', 0, 1, 'BlogPost', '2020-12-31T00:31:34', '', '2020-12-31T00:31:34', '48v9vt5vxznr5z9m1df9zmvjm8', 'admin', 'The title') ; select last_insert_rowid() [Id];update [ContentItemIndex] set [DocumentId] = 19 where [Id] = (last_insert_rowid());insert into [ContainedPartIndex] ([ListContentItemId], [Order]) values ('491emynv0kavbzhy40xmqv1wds', 0) ; select last_insert_rowid() [Id];update [ContainedPartIndex] set [DocumentId] = 19 where [Id] = (last_insert_rowid());insert into [AutoroutePartIndex] ([ContentItemId], [Path], [Published], [Latest], [ContainedContentItemId], [JsonPath]) values ('4cpw0fnmjb1kp07dmzxx8n8ecg', 'blog/the-title', 0, 1, '', '') ; select last_insert_rowid() [Id];update [AutoroutePartIndex] set [DocumentId] = 19 where [Id] = (last_insert_rowid());
When we create a blog post there aren't any delete requests, because there is nothing before but when we update we need to delete all the indexes and these calls contains unnecessary deletes because for example there is no LayerMetaDataIndex on the blog post but the logic here is hey, we are dealing with a content item and there is an index for all of the content items called LayerMetadataIndex. So it will send the delete just in case which is dumb. But you can see here for each index we have a delete and it used to be a single query communication query with the database. Now let's divide by two the part where we insert and update the indexes. Instead of having to insert something then update something for each index now, we have just inserted it. We still have all the deletes and some of them are still useless but these are quick. So we have just this request now when we update a single blog post.
insert into [Document] ([Id], [Type], [Content], [Version]) values (23, 'OrchardCore.ContentManagement.ContentItem, OrchardCore.ContentManagement.Abstractions', '{"ContentItemId":"4cpw0fnmjb1kp07dmzxx8n8ecg","ContentItemVersionId":"4ypdrxm7xbndr0dvcpwaraa95g","ContentType":"BlogPost","DisplayText":"The title","Latest":true,"Published":false,"ModifiedUtc":"2020-12-31T05:55:56.8113646Z","PublishedUtc":"2020-12-31T01:22:37.7926461Z","CreatedUtc":"2020-12-31T00:31:34.3346095Z","Owner":"48v9vt5vxznr5z9m1df9zmvjm8","Author":"admin","TitlePart":{"Title":"The title"},"AutoroutePart":{"Path":"blog/the-title","SetHomepage":false,"Disabled":false,"RouteContainedItems":false,"Absolute":false},"BlogPost":{"Subtitle":{"Text":"Subtitle"},"Image":{"Anchors":[],"Paths":[],"MediaTexts":[]},"Tags":{"TagNames":["Space"],"TaxonomyContentItemId":"4ykev5wxfcny7tvsahz9y64mwe","TermContentItemIds":["4nv0z7r24r1vw3sfpq7t6xws59"]},"Category":{"TaxonomyContentItemId":"4tpy2wv97bkbf0zkx8tyd1bm4q","TermContentItemIds":["4bsstr09f29rp0sgy85n9f07wj"]}},"MarkdownBodyPart":{"Markdown":"Some text"},"ContainedPart":{"ListContentItemId":"491emynv0kavbzhy40xmqv1wds","Order":0}}', 1);delete from [ContentItemIndex] where [DocumentId] = 22;delete from [AliasPartIndex] where [DocumentId] = 22;delete from [LayerMetadataIndex] where [DocumentId] = 22;delete from [ContainedPartIndex] where [DocumentId] = 22;delete from [AutoroutePartIndex] where [DocumentId] = 22;delete from [TaxonomyIndex] where [DocumentId] = 22;insert into [ContentItemIndex] ([ContentItemId], [ContentItemVersionId], [Published], [Latest], [ContentType], [ModifiedUtc], [PublishedUtc], [CreatedUtc], [Owner], [Author], [DisplayText], [DocumentId]) values ('4cpw0fnmjb1kp07dmzxx8n8ecg', '4q3271jp1705etwnf52c0nnbwz', 1, 0, 'BlogPost', '2020-12-31T01:22:37', '2020-12-31T01:22:37', '2020-12-31T00:31:34', '48v9vt5vxznr5z9m1df9zmvjm8', 'admin', 'The title', 22) ; select last_insert_rowid() [Id];insert into [ContainedPartIndex] ([ListContentItemId], [Order], [DocumentId]) values ('491emynv0kavbzhy40xmqv1wds', 0, 22) ; select last_insert_rowid() [Id];insert into [AutoroutePartIndex] ([ContentItemId], [Path], [Published], [Latest], [ContainedContentItemId], [JsonPath], [DocumentId]) values ('4cpw0fnmjb1kp07dmzxx8n8ecg', 'blog/the-title', 1, 0, '', '', 22) ; select last_insert_rowid() [Id];insert into [TaxonomyIndex] ([TaxonomyContentItemId], [ContentItemId], [ContentType], [ContentPart], [ContentField], [TermContentItemId], [DocumentId]) values ('4ykev5wxfcny7tvsahz9y64mwe', '4cpw0fnmjb1kp07dmzxx8n8ecg', 'BlogPost', 'BlogPost', 'Tags', '4nv0z7r24r1vw3sfpq7t6xws59', 22) ; select last_insert_rowid() [Id];insert into [TaxonomyIndex] ([TaxonomyContentItemId], [ContentItemId], [ContentType], [ContentPart], [ContentField], [TermContentItemId], [DocumentId]) values ('4tpy2wv97bkbf0zkx8tyd1bm4q', '4cpw0fnmjb1kp07dmzxx8n8ecg', 'BlogPost', 'BlogPost', 'Category', '4bsstr09f29rp0sgy85n9f07wj', 22) ; select last_insert_rowid() [Id];update [Document] set [Content] = '{"ContentItemId":"4cpw0fnmjb1kp07dmzxx8n8ecg","ContentItemVersionId":"4q3271jp1705etwnf52c0nnbwz","ContentType":"BlogPost","DisplayText":"The title","Latest":false,"Published":true,"ModifiedUtc":"2020-12-31T01:22:37.6390174Z","PublishedUtc":"2020-12-31T01:22:37.7926461Z","CreatedUtc":"2020-12-31T00:31:34.3346095Z","Owner":"48v9vt5vxznr5z9m1df9zmvjm8","Author":"admin","TitlePart":{"Title":"The title"},"AutoroutePart":{"Path":"blog/the-title","SetHomepage":false,"Disabled":false,"RouteContainedItems":false,"Absolute":false},"BlogPost":{"Subtitle":{"Text":"Subtitle"},"Image":{"Anchors":[],"Paths":[],"MediaTexts":[]},"Tags":{"TagNames":["Space"],"TaxonomyContentItemId":"4ykev5wxfcny7tvsahz9y64mwe","TermContentItemIds":["4nv0z7r24r1vw3sfpq7t6xws59"]},"Category":{"TaxonomyContentItemId":"4tpy2wv97bkbf0zkx8tyd1bm4q","TermContentItemIds":["4bsstr09f29rp0sgy85n9f07wj"]}},"MarkdownBodyPart":{"Markdown":"Some text"},"ContainedPart":{"ListContentItemId":"491emynv0kavbzhy40xmqv1wds","Order":0}}', [Version] = 1 where [Id] = 22;insert into [ContentItemIndex] ([ContentItemId], [ContentItemVersionId], [Published], [Latest], [ContentType], [ModifiedUtc], [PublishedUtc], [CreatedUtc], [Owner], [Author], [DisplayText], [DocumentId]) values ('4cpw0fnmjb1kp07dmzxx8n8ecg', '4ypdrxm7xbndr0dvcpwaraa95g', 0, 1, 'BlogPost', '2020-12-31T05:55:56', '2020-12-31T01:22:37', '2020-12-31T00:31:34', '48v9vt5vxznr5z9m1df9zmvjm8', 'admin', 'The title', 23) ; select last_insert_rowid() [Id];insert into [ContainedPartIndex] ([ListContentItemId], [Order], [DocumentId]) values ('491emynv0kavbzhy40xmqv1wds', 0, 23) ; select last_insert_rowid() [Id];insert into [AutoroutePartIndex] ([ContentItemId], [Path], [Published], [Latest], [ContainedContentItemId], [JsonPath], [DocumentId]) values ('4cpw0fnmjb1kp07dmzxx8n8ecg', 'blog/the-title', 0, 1, '', '', 23) ; select last_insert_rowid() [Id];
Using the other PR we are able to teach the index when not to send deletes as there will never be any index related to this Document, so don't send deletes. After that, you can see there is no more LayerMetaDataIndex and no more AliasPartIndex calls because the blog post doesn't have these.
delete from [ContentItemIndex] where [DocumentId] = 23;delete from [ContainedPartIndex] where [DocumentId] = 23;delete from [AutoroutePartIndex] where [DocumentId] = 23;delete from [TaxonomyIndex] where [DocumentId] = 23;
This is by having a new method on the IndexProvider to explain when to not use the IndexProvider. So it would not even go to the Map method. In this case, we say if you don't have the AliasPart, don't use the IndexProvider.
context.For<AliasPartIndex>().When(c => c.Has<AliasPart>()).Map(contentItem =>
Check out the following recording on YouTube to know more about this YesSql improvement!
News from the community
Lombiq Hosting - Azure Application Insights
This new Orchard Core module from Lombiq enables easy integration of Azure Application Insights telemetry into Orchard. Just install the module, configure the instrumentation key from a configuration source (like the appsettings.json file) as normally for AI, and collected data will start appearing in the Azure Portal. Would like to learn more about our new module? Then head to the repository now where you can find every detail about how to set up and use that module in your site!
Orchard Dojo Newsletter
Lombiq's Orchard Dojo Newsletter has 191 subscribers! We have started this newsletter to inform the community around Orchard with the latest news about the platform. By subscribing to this newsletter, you will get an e-mail whenever a new post published to Orchard Dojo, including This week in Orchard of course.
Do you know of other Orchard enthusiasts who you think would like to read our weekly articles? Tell them to subscribe here!