# "Go doing things Rust can't" or unsizing unsized values Consider the code in [the go tour](https://go.dev/tour/methods/16), showing `interface{}`: [^2] ```go package main import "fmt" func doit(i interface{}) { switch v := i.(type) { case int: fmt.Printf("Twice %v is %v\n", v, v*2) case string: fmt.Printf("%q is %v bytes long\n", v, len(v)) default: fmt.Printf("I don't know about type %T!\n", v) } } func main() { doit(21) doit("hello") doit(true) } ``` We can interrogate an `interface {}` value about its type, then downcast to the concrete type and do something with the value. Naïvely, you would think that we can do the same thing in Rust with the `Any` trait, but there are subtle pitfalls. Let's start thinking about what `doit` should look like. We want to consume a value, but to produce a single function we should not use generic arguments, but some kind of `dyn`-object. This is important in some areas, where generic arguments are simply not acceptable; For example to allow a trait to be `dyn`-compatible, or if `doit` is supposed to be an `extern` function, or to pass the function around as a `fn`/`Fn` object. ```rust= use std::any::Any; fn doit(i: &dyn Any) { if let Some(v) = i.downcast_ref::<i32>() { println!("Twice {v} is {}", v*2); } else if let Some(s) = i.downcast_ref::<String>() { println!("String: {s} is {} bytes long", s.len()); } else if let Some(s) = i.downcast_ref::<&'static str>() { println!("&'static str: {s} is {} bytes long", s.len()); } else { println!("I don't know about type {:?}", i.type_id()); } } ``` Specifically, the second call above, `doit("hello")` turns out to be problematic. ```rust=+ pub fn main() { doit(&21); doit(&"hello".to_string()); doit(&true); } ``` We allocate and copy the string and that is really annoying! We could alternatively downcast to `&'static str` and add a case. ```rust if let Some(s) = i.downcast_ref::<&'static str>() { println!("{s} is {} bytes long", s.len()); } // ... doit(&"hello"); ``` But notice the `'static` lifetime - which we deviously could omit and write as simply `&str` - that forces us to either leak the string or have it be a constant for the program duration. But accepting a string as `&str` where the borrow lives for just some duration is not possible. ```rust fn doit_for_str(s: &str) { // error[E0277]: the size for values of type `str` cannot be known at compilation time doit(s); // ^ doesn't have a size known at compile-time // help: consider borrowing the value, since `&&str` can be coerced into `&(dyn Any + 'static)` // // but this suggestion does not work! // // error: lifetime may not live long enough doit(&s); // ^^ coercion requires that `'1` must outlive `'static` } ``` # Reflecting So what happens? When implementing the coercion the compiler has to synthesize the pointer metadata for `dyn Any`, which happens to be a `&'static VTable` where `VTable` deserves a second look shortly. Since this vtable lives in `.rodata`, the compiler has to derive it from the type we coerce from - `str` in this case - at compile time and can not use the value. In other words, you can only coerce `Sized` values to a `dyn`-object in current Rust. Diving into compiler internals for a moment - that we will rely on later - the vtable contains four items, layed out as if in a `#[repr(C)]` struct in this order: - the drop glue function, roughly of the signature `fn(*const ())`. Since dropping a `str` is trivial, this could be a no-op here, - the size of the value. Uh oh, this one is problematic. Since the string could have any size (length) at runtime, this one can not be derived at compile time, - the align of the value. This is again unproblematic and can be derived from the type - align is `1`, - the `type_id` function, roughly again of signature `fn(*const ()) -> TypeId`. I might have smuggled it past you, dear reader, in the drop function, but these function pointers do not receive the original pointer metadata! When the compiler synthesizes the call `<dyn Any as Any>::type_id(i)`, it will use the pointer metadata to get this function pointer, then strip off the metadata and pass only the thin pointer. Since we do not need to inspect the `str` to derive the `TypeId`, we dodge complications again. In general though, the original pointer metadata (i.e. the string length) is lost. [^1] # Hacking We will have to do the vtable construction "by hand". Since we *have* to construct it at runtime because we need it to store the actual size of the string, we need a place to store this vtable on stack. ```rust=+ use std::any::TypeId; use std::mem::MaybeUninit; #[repr(C)] struct AnyVTable { drop: fn(*const()), size: usize, align: usize, type_id: fn(*const()) -> TypeId, } pub struct Host { vtable: MaybeUninit<AnyVTable>, } impl Host { pub fn new() -> Self { Host { vtable: MaybeUninit::uninit() } } pub fn borrow<'a>(&'a mut self, s: &'a str) -> &'a dyn Any { todo!() } } ``` Our goal will be implementing the `borrow` function, which already contains a small lie. The returned reference can only be used for `'a` which ensures that the vtable which we store in the `Host` is also still borrowed. But with a bit of nightly magic, we can extract the pointer metadata (with [`std::ptr::metadata()`](https://doc.rust-lang.org/std/ptr/fn.metadata.html)) which forgets about the lifetime and can be used past the borrow. I would propose to add a lifetime for the vtable after the `dyn` keyword that would fix this. ```rust impl Host { pub fn borrow<'a, 's>(&'s mut self, s: &'a str) -> &'a dyn<'s> Any { } // ^^^^ // `dyn` could default like other lifetime to dyn<'static>, which would // have the same meaning as now, in some contexts and introduce a fresh // lifetime in other contexts (function signatures). I'm not certain this // default is necessarily what we want or need. // In this case, the above would omit the 's lifetime by the way lifetimes // in return types are inferred to default to the lifetime of `&mut self`. // Perhaps for compat, the default should always be 'static and the explicit // form above is better. } ``` But I digress, suspend your disbelief a bit. Let's try this ```rust=+ impl Host { pub fn borrow<'a>(&'a mut self, s: &'a str) -> &'a dyn Any { let vtable = self.vtable.write(AnyVTable { drop: |_| (), // str has trivial drop size: s.len(), align: 1, type_id: |_| TypeId::of::<str>(), }); #[repr(C)] struct FatPtr<'a> { thin: *const (), vtable: &'a AnyVTable, } let rep = FatPtr { thin: s.as_ptr() as *const (), vtable, }; unsafe { std::mem::transmute(rep) } } } fn doit_str(s: &str) { let mut host = Host::new(); doit(host.borrow(s)); } ``` It runs and produces the following output ```! I don't know about type TypeId(0xb7381ee5f3fdfc9d7fa709f37e151622) ``` Ah yes, because the type id is that of `str` now, none of the cases above work. We need a way to downcast again. If we try the `downcast_ref` method, we come across another obstacle, mainly `downcast_ref<T>` requires `T: Sized`. This is because of a problem we discovered earlier. Since the call through the vtable throws away the pointer metadata, in general there is no way to recover it and the downcast can only work for types that have no pointer metadata, i.e. that are `Sized`. But alas, for `str` in particular we can actually recover all we need, specifically the length of the string from the metadata we are given. ```rust=+ fn downcast_dyn_str(s: &dyn Any) -> Option<&str> { if s.type_id() != TypeId::of::<str>() { return None; } // On nightly we can use std::str::from_raw_parts Some(unsafe { std::str::from_utf8_unchecked( std::slice::from_raw_parts( std::ptr::from_ref(s).cast(), size_of_val(s), ) ) }) } // Add the case to doit: if let Some(s) = downcast_dyn_str(i) { println!("str: {s} is {} bytes long", s.len()); } ``` With that, finally we can witness ```rust doit_str("hello"); // str: hello is 5 bytes long ``` # Miri If we run [the code](https://play.rust-lang.org/?version=stable&mode=debug&edition=2024&gist=0facbb927c95bebb75a16af9d8b5d200) under miri for verification, it will claim ```! error: Undefined Behavior: constructing invalid value of type &dyn std::any::Any: encountered 0x39858[alloc4896]<19017>, but expected a vtable pointer ``` which makes sense, since we definitely forged a vtable pointer that didn't get blessed and is unholy. This is expected. What I want to claim though is that the compiler could in theory do the transformation for us, with one small caveat. We need to revisit the pointer metadata again. If the functions in the vtable would receive the full fat pointer, we could for example stuff the "previous" metadata in there. For our use case, suppose `Host` didn't simply contain a `AnyVTable` but a struct like ```rust struct MetaChained { prev: usize, // str length in our case vtable: AnyVTable, } ``` Now, a fat pointer would still contain a pointer to the `AnyVTable`, i.e. the second field of the struct, but with some quick pointer arithmetic we could recover the previous pointer metadata. Vtables that are instantiated at compile time could strip the metadata as they do right now, but instead of doing that before the call into the vtable function, they would do so inside. This would allow `downcast_ref::<T: ?Sized>()`, too. # Conclusion The sense of all of this is for you to judge. I have [a repo with further experiments](https://github.com/WorldSEnder/unsize-the-unsize), which generalizes the above for a `Host` to coerce any `T` to `U` under an assumption equivalent to a generalized `T: CoerceUnsized<U>`. In particular you can coerce `Box<str>` to `Box<dyn Any>` just as you would coerce references. Bar a language change that allows annotations with an additional lifetime such as `dyn<'a> Any` this is declared `unsafe` to prevent accidental misuse though. The experiment can not support other coercions such as `[T]` to `dyn Any` without fat pointers getting passed through to the vtable functions. For now, it allows `T: Copy` where the drop is trivial and we do not need to recover the slice metadata. The same restriction of losing the metadata prevents extending to other less trivial traits, but it is the only thing preventing us from recovering the correct receiver value. I do not know the right people to land this. I do hope though that at some point in the future I can at least coerce `&[T]` to `&dyn Any` and `downcast_ref` back again natively. [^1]: for interested people, the metadata stripping is the raison d'être of the compiler synthesized [`DispatchFromDyn`](https://doc.rust-lang.org/std/ops/trait.DispatchFromDyn.html) trait. [^2]: `do` has been renamed to `doit` because the former is a reserved keyword.