### Summary
- `GenericEntityPersistence` is now a type of `GenericPersistence`. This allows developers to serialize both entities and non-entities in the same system. 🎉
- Each `SerializationThreadWorker` now allocates 1MB of heap for serialization _permanently_. If more memory is needed, that thread will double it's memory, not to exceed increments of 64MB.
- Several bugs with serialization introduced with the pure MMF implementation have been fixed.
- `BinaryFileReader` has been added back. 🎉
- Adds `world.useMultithreadedSaves` to allow disabling threaded saves.
> [!IMPORTANT]
> **Developer Note**
> The split file serialization has been deprecated and is no longer used. We have effectively gone back to the same file writing we had before the pure MMF implementation.
### Summary
Updates the serialization strategy to use `MemoryMappedFile` instead of thick buffers. This has the benefit of being on-par with the current implementation (based on hardware/OS), however won't incur the double-memory issue.
> [!Important]
> **Developer Note**
> The `BinaryFileWriter` and `BinaryFileReader` has been removed in favor of `MemoryMapFileWriter` and `UnmanagedDataReader`
## Summary
### Changes
- Adds `[IgnoreDupe]` and `[SerializedIgnoreDupe]`
- Updates all _known_ classes that need the attribute. Some might be missing, please helps us find them!
- Adds `Item.Dupe()` command and encapsulates `CopyProperties` and `OnAfterDuped`. This is also overridable.
- Updates Dupe command to use the new logic.
- Fixes duping multiple kinds of objects that used to be outright broken.
### Bug Fixes
- Fixes issue with durability after duping
- Fixes issue with hue after duping
> [!Note]
> **Developer Note**
> Customizing how duping an item works now requires two steps:
> 1. Add `[IgnoreDupe]` or `[SerializedIgnoreDupe]` to the property/field
> 2. Add custom logic in an `OnAfterDuped` override
>
> When do you need to do this?
> *When the property being copied is not a primitive, and you need to manually deep-clone the contents of the property such as with Lists, Dictionaries, or sub classes.*
### Summary
Adds a generic entity persistence. This can be used to create new entity types that have a `Serial`.
Here is an example:
```cs
public class BOBEntries : GenericEntitySerialization<IBOBEntry>
{
public static void Configure()
{
Configure("BOBEntries");
}
}
```
The annotation tells the system what folder to serialize the entries to. The class/interface (`IBOBEntry`) is the root type that implements `ISerializable`.
## MAJOR CHANGE
Added a champion title system to facilitate the existing champion titles. This should make it easier to extend or create other related game content. Champion titles will be saved in a folder called _ChampionTitles_.
### Motivation
The motivation to refactor was two-folder, but mostly related to performance in two ways.
First, every player had a ChampionTitleInfo object with an array of ChamptionTitleInfo. We want to eliminate the need for this information unless a player actually uses it. This should save a considerable amount of memory.
Second, to facilitate the atrophy mechanic, the champion titles would run atrophy post-world save, adding to the time that the server is frozen. Eliminating this post-world save side effect unlocks our ability to further optimize the world save process since there are no direct side effects.
### Bugs fixed
- [X] Fixed titles getting cut off on the paperdoll
- [X] Fixed champion title not displaying overhead (OPL)
### Screenshots
<img width="216" alt="image" src="https://github.com/modernuo/ModernUO/assets/3953314/8916f895-8d68-4fb0-892e-108a0c43be90">
### Changes
* Implements an AfterSerialize method that is executed synchronously.
* Removes `BeforeSerialize` support since it was dangerous in its current implementation.
* Moves PlayerMobile kill/virtual decay to AfterSerialize.
* Adds kill decay to after Deserialize.
## Changes
* Improves type hashing by introducing xxHash3 (64bit)
* Removes individual `tdb` files in favor of a single `SerializedTypes.db` file. This file is only used to identify a type that is being deserialized, which doesn't exist.
* Adds duplicate type alias detection
* Adds `AssemblyHandler.FindTypeByHash`
View changed files whitespaces: https://github.com/modernuo/ModernUO/pull/1172/files?diff=split&w=1
## SerializedTypes.db
The serialized types file is used to get back the original name of a type in case it no longer exists in code. This can easily be necessary if a class is renamed in code and no `TypeAlias` is provided.
### Format
byte[4] - version
byte[4] - count
--array--
byte[8] - xxHash
byte[1] - flag, 0 - null, 1 - not null
byte[n] - Full class name in UTF8
### Example
<img width="472" alt="SerializedTypes_Example" src="https://user-images.githubusercontent.com/3953314/195255429-31d24293-6bd1-419e-811b-07874dd0f78d.png">
## Benchmarks
Serialized 500 Type fields. The 8192bytes comes from the _ConcurrentQueue_ that would later be used for SerializedTypes.
Note that the queue is never cleared, so it's size grew considerably.
```cs
| Method | Mean | Error | StdDev | Allocated |
|--------------------- |---------:|---------:|---------:|----------:|
| BenchmarkXXHash | 18.44 us | 0.278 us | 0.260 us | 8192 B |
| BenchmarkTypeStrings | 25.09 us | 0.292 us | 0.259 us | - |
```
TODO:
* Add support in the Serialization Generator for `ReadType()` and `Write(Type)`
* Remove `SetTypeRef` from Serialization Generator
**Only one functional change**
* Fixes a bug in LogFactory where `Warning` is being logged as `Information`
Non-functional changes:
* Updates/Fixes copyright headers
* Removes namespace scopes for core files.
View with [whitespace off](https://github.com/modernuo/ModernUO/pull/1187/files?w=1).
* Adds `BeforeSerialized` for entities to handle cleanup.
* Adds `Created` and `LastSerialized` fields to all entities. While this is a big bloat, this will be necessary for identifying dangling references to other invalid entities.
* Changes formula for determining a valid reference to be _not null, not deleted, and reference's created date must be at or before the entities last serialized date_.
* Adds versioning to idx file and serializes `Created` and `LastSerialized`.
* Fixes Save Stats and also disables it by default.
- Adds `[ManualDirtyChecking]` for anyone that adds it themselves.
- Adds detection of `[Serializable]` and `[ManualDirtyChecking]` at startup to help identify scripts that need migration.
### Additions
- Automatically opts-out `Item/Mobile/Guild/Accounts` from dirty checking with a new property `UseDirtyChecking`
- Codegen now enables `UseDirtyChecking` via getter. This requires that the property is `virtual` for derived types.
### Fixes
- Fixes `EncodedInt` being broken
- Fixes issues with new custom serializable types that are not derived from Item/Mobile/etc.
- Removes double dirty checking.
### Example of a brand new serializable type that isn't an Item/Mobile/etc.
User created code:
```cs
using System;
namespace Server.Items
{
[Serializable(0)]
public partial class NewTestEntityObject : ISerializable
{
[EncodedInt]
[SerializableField(0)]
private int _someProperty;
public NewTestEntityObject()
{
SetTypeRef(GetType());
// Add to serial tracking like World.Item
/*
Serial = World.NewEntity;
World.AddEntity(this);
*/
}
[AfterDeserialization]
private void AfterDeserialization()
{
Console.WriteLine("This ran!");
}
public int TypeRef { get; }
public Serial Serial { get; }
public void Delete()
{
}
public bool Deleted { get; set; }
public void SetTypeRef(Type type)
{
// Type tracking for persistence goes here
/*
TypeRef = World.NewEntityTypes.IndexOf(type);
if (TypeRef == -1)
{
World.NewEntityTypes.Add(type);
TypeRef = World.NewEntityTypes.Count - 1;
}
*/
}
}
}
```
Generated code:
```cs
namespace Server.Items
{
public partial class NewTestEntityObject
{
#pragma warning disable 0414
private const int _version = 0;
#pragma warning restore 0414
public int SomeProperty
{
get => _someProperty;
set
{
if (value != _someProperty)
{
_someProperty = value;
((ISerializable)this).MarkDirty();
}
}
}
long ISerializable.SavePosition { get; set; } = -1;
BufferWriter ISerializable.SaveBuffer { get; set; }
bool ISerializable.UseDirtyChecking => true;
public NewTestEntityObject(Serial serial)
{
Serial = serial;
SetTypeRef(typeof(NewTestEntityObject));
}
public void Serialize(IGenericWriter writer)
{
writer.WriteEncodedInt(_version);
writer.WriteEncodedInt(SomeProperty);
}
public void Deserialize(IGenericReader reader)
{
var version = reader.ReadEncodedInt();
SomeProperty = reader.ReadEncodedInt();
Timer.DelayCall(AfterDeserialization);
}
}
}
```
### Features
* Fully abstracts serialization by using compile-time attributes.
* Supports serializing the following:
- Primitives (integers, strings, etc)
- IP Addresses
- BigDecimal
- DateTime, Delta DateTimes
- TimeSpan
- Server.Race
- Server.Map
- Point2D, Point3D, Rect2D, Rect3D
- Existing/New `ISerializable` references
- Lists/Sets of serializable types
- Type with a `Serialize` method and constructor that takes an `IGenericReader`
* Supports forward-only migration
* Supports existing RunUO deserialization for older versions by changing to the following signature:
- `public void OldDeserialize(IGenericReader reader, int version)`
- Must remove deserializing the version since this is already done
* Supports serializing from private fields or custom made properties.
* Types do not require inheriting Item/Mobile. Code gen will fully create `ISerializable` information.
- This is not recommended yet, since it requires wiring to `Persistence` which will cause lots of unresolved symbol errors until code gen is built.
### Example
```cs
using System.Collections.Generic;
namespace Server.Items
{
[Serializable(1)]
public partial class TestItem1 : Item
{
[SerializableField(1)]
[SerializableFieldAttr("[CommandProperty(AccessLevel.Administrator)]")]
private List<Item> _someProperty;
private void Deserialize(IGenericReader reader, int version)
{
}
}
}
```
Generates this:
```cs
namespace Server.Items
{
public partial class TestItem1
{
#pragma warning disable 0414
private const int _version = 1;
#pragma warning restore 0414
[CommandProperty(AccessLevel.Administrator)]
public System.Collections.Generic.List<Server.Item> SomeProperty
{
get => _someProperty;
set
{
if (value != _someProperty)
{
((ISerializable)this).MarkDirty();
_someProperty = value;
}
}
}
public TestItem1(Serial serial) : base(serial)
{
}
public override void Serialize(IGenericWriter writer)
{
var savePosition = ((Server.ISerializable)this).SavePosition;
if (savePosition > -1)
{
writer.Seek(savePosition, System.IO.SeekOrigin.Begin);
return;
}
writer.WriteEncodedInt(_version);
writer.Write(_someProperty);
}
public override void Deserialize(IGenericReader reader)
{
var version = reader.ReadEncodedInt();
if (version < 1)
{
OldDeserialize(reader, version);
((Server.ISerializable)this).MarkDirty();
return;
}
SomeProperty = reader.ReadEntityList<Server.Item>();
}
}
}
```
And this:
```json
{
"version": 1,
"type": "TestItem1",
"properties": [
{
"name": "SomeProperty",
"type": "System.Collections.Generic.List\u003CServer.Item\u003E",
"rule": "ListMigrationRule",
"ruleArguments": [
"Server.Item",
"SerializableInterfaceMigrationRule"
]
}
]
}
```
- [X] Removes some uses of linq
- [X] Creates struct based enumerator for Skills
- [X] Creates a struct based enumerator for TypeCache
- [X] Changes HarvestDefinition to an init array instead of List
- [X] Changes HeritageTokenGump Response from List to Array
- [X] Fixes an issue where a buffer smaller than 8 bytes would not double with enough space in some cases.
- [X] Fixes an issue with dupe copying the savebuffer reference (ugh).
- [X] Streamlines the IGenericWriter API to use better generics.
- [X] Streamlines the IGenericReader API to use better generics.
- [X] Forces `tidying` of a List/HashSet to be done externally since Writers/Readers should not have side effects.
- [X] Fixes an issue where Tidying a list didn't TrimExcess, causing memory leaks.
- [X] Reverted the meaning of `World.Running` to specifically refer to any world state post world loading.
- NOTE: Do not use this if you want to block on world saves. Instead use checks against `WorldState.Saving` states.
- [X] Fixes an issue with serializing negative DateTime deltas.
- [X] Fixes a potential issue with serializing non-UTC DateTime.
Bumps release version
- [X] Removes some string allocations (e.g. split)
- [X] Optimizes some collections
- [X] Converts insensitive to extension methods of built-ins.
- [X] Adds ordinal (case sensitive) string helpers
- [X] Fixes conditionals for in-game commands so they use Ordinal comparisons.
- [X] Replaces ToLower.Contains with InsensitiveContains
- [X] Adds ValueStringBuilder
- [X] Implements ValueStringBuilder in a few places where it makes sense
- [X] Removes the redundant Wrap function and replaces it with an optimized version
- [X] Fixes list conversions in Utility
Closes#351
Bumps release version
- [X] Fixes bugs in the buffer writer
- [X] Removes background save commands
- [X] Avoids calling Serialize() on deserialization
- In a future optimization I will copy the entire object to a buffer, deserialize using a SpanReader, then use that buffer for the SaveBuffer
- [X] Fixes issue with decay queue.
Bumps release version
- [X] Removes all save strategies
- [X] Removes duplicate file writer that won't be used
- [X] Removes persistence (it will become a duplicate system)
- [X] Add a save position variable to skip serializing a clean item/mobile
- [X] Update Guilds/Accounts to be IEntity types
- Because guilds are abstract, this may make serialization tricky.
- [X] Update the generalized IEntity writing
- [X] Update the load/save to write to buffers then to files in background
Bumps release version
* Fixes a bug with the main loop. Removes unnecessary optimizations for RDRand.
* WIP - Rewriting packets.
* Cleanup and updates README
* Converts more packets
* Fixes header
* Converts Effects packets.
* Forgot the send command
* Adds the start of containers packets.
* Adds message packets with caching
* Extends the send methods
* Starts to add Acquire methods
* Converted the packets
* Cleanup
* Cleanup
* Adds more packets
* Adds more packets
* Fixes clearing arrays from pool. Adds Mobile Incoming
* Converts more packets.
* Moves more packets
* Moves more packets
* Test compression
* Merges
* Migrating to an idiomatic syntax that also supports compression, and proxying.
* Optimize the stack alloc. Changes attributes
* Converted container packets
* Visual Studio doesn't auto save files. I am still getting used to subpar IDEs.
* Adds static packet caching. Will profile later. Converts more packets
* Fixes packets and changes how compression is configured
* WIP - Adds basics for Gumps. Not finished though.
* Fix file
* Fixes formatting
* Changed SpanWriter to be more idiomatic.
* Fixes Span vs RawSpan and missing stackallocs
* Converts over some more gumps
* WIP - Deletes 32bit support.
* Drops RDRand32 support.
* Fixes gump compilation
* Converting gump components
* Removes old huffman compression function
* Revert signature for backwards compatibility
* WIP - Converting more gump components
* Creates ArraySet for the strings. Updates AppendTo to reference that.
* Converts the maining gump components
* Cleans up gump components
* Cleans up directives
* Cleans up ArraySet and moves it. Adds null-coalescing-assignment
* Cleanup, Target Packets, and C# 8 changes.
* Removed OPL Packet
* World packets
* Fixing packet uses
* Cleans up code. Fixes packet uses in various places.
* More cleanup for packets
* Code cleanup
* Converts more packet uses and cleans up more code
* More code cleanup
* Finishes fixing the packets in Item
* Updates secure trade packets
* Updates core and gets it to compile.
* Moved packets to scripts. Fixed account handler use of packets
* Code cleanup
* Updates chat packets
* Code cleanuo
* Adds party packets, but need to implement them.
* Party packets WIP
* Finishes party packets
* Rearrange movement namespaces and classes
* Finishes plant packets
* Code Formatting
* Fixes moving effects
* Code cleanup, eliminates equipinfo
* Adds more packets. Fixes bugs with various packets.
* Finishes mahjon packets
* Fixes mahjong packet
* Code cleanup
* Finishes mahjong packets
* Adds Map packets
* Add multifacet maps and charts
* Cleans up some packets with UTF8
* Optimizes packets
* Cleans up more packets. Moves the MessageHelper
* Removes assistant support. Removes extended protocol. Incorporates MapUO packets as normal packets.
* Updates protocol extensions packet receiver
* Fixes a few bugs. Fixes a few more packets.
* Code cleanup and fixing more packets
* Fixes packet effects
* Cleaned up more code
* Buff Icon cleanup
* Removed unused constructors
* Code Cleanup. Adds BoatHS Packets
* Moves house files. Updates house foundation packets.
* Deployment cleanup
* Fixes:
* Code cleanup
* More code cleanup
* More code cleanup
* Cleaned up BaseHouse
* Enforces styling
* Converts foreach to linq where possible.
* Dont need that
* Goals/Readme updates
* Removes 32bit support at the highest level. Turns on HRT by default.
* Code cleanup. Fixes extended features packet.
* Fixes various bugs
* Code cleanup
* Code cleanup
* Code cleanup. Fixes gump X/Y assignment.
* Code cleanup using |= operator
* More code cleanup
* Cleanup
* Fixes spacing issues. Thanks Visual Studio. You suck.
* Compiler error
* Fixes NPE from RunUO 2.7
* Renames ScriptCompiler to AssemblyHandler. Fixes packets. Updates README
* Fixes more packets. Stupid trailing nulls.
* Fixes various bugs.
* Fixes for gumps
* Fixes more gump stuff. Going to split it out later since it is getting insane
* Recoded the gump writing
* WIP
* *Added output path of scripts project to dev branch
*Activated debugging in code
