Skip to content

Commit

Permalink
small changes
Browse files Browse the repository at this point in the history
  • Loading branch information
ArthurZucker committed Jun 14, 2024
1 parent 5540136 commit ba03c16
Show file tree
Hide file tree
Showing 2 changed files with 8 additions and 7 deletions.
13 changes: 7 additions & 6 deletions tokenizers/display_derive/src/vendored.rs
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
use std::{env::VarError, error::Error};

Check warning on line 1 in tokenizers/display_derive/src/vendored.rs

View workflow job for this annotation

GitHub Actions / Check it builds for Windows 32-bit (3.9)

unused imports: `env::VarError`, `error::Error`

Check warning on line 1 in tokenizers/display_derive/src/vendored.rs

View workflow job for this annotation

GitHub Actions / Check it builds for Windows 32-bit (3.10)

unused imports: `env::VarError`, `error::Error`

Check warning on line 1 in tokenizers/display_derive/src/vendored.rs

View workflow job for this annotation

GitHub Actions / Check it builds for Windows 32-bit (3.7)

unused imports: `env::VarError`, `error::Error`

Check warning on line 1 in tokenizers/display_derive/src/vendored.rs

View workflow job for this annotation

GitHub Actions / Check it builds for Windows 32-bit (3.8)

unused imports: `env::VarError`, `error::Error`

Check warning on line 1 in tokenizers/display_derive/src/vendored.rs

View workflow job for this annotation

GitHub Actions / Check everything builds

unused imports: `env::VarError`, `error::Error`

use crate::parsing;
use proc_macro2::TokenStream;
use quote::{format_ident, ToTokens};
use syn::{
parse::{Parse, ParseStream},
punctuated::Punctuated,
token,
Expr,
token, Expr,
};

/// Representation of a [`fmt`]-like attribute.
Expand All @@ -32,19 +33,19 @@ pub struct FmtAttribute {

impl Parse for FmtAttribute {
fn parse(input: ParseStream<'_>) -> syn::Result<Self> {

Ok(Self {
let attribute = Self {
lit: input.parse()?,
comma: input
.peek(token::Comma)
.then(|| input.parse())
.transpose()?,
args: input.parse_terminated(FmtArgument::parse)?,
})
};
println!("Parsing FMTAttribute, {}, ",attribute.lit.token().to_string());
Ok(attribute)
}
}


impl ToTokens for FmtAttribute {
fn to_tokens(&self, tokens: &mut TokenStream) {
self.lit.to_tokens(tokens);
Expand Down
2 changes: 1 addition & 1 deletion tokenizers/src/decoders/byte_fallback.rs
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
use crate::tokenizer::{Decoder, Result};
use monostate::MustBe;
use display_derive::Display;
use monostate::MustBe;
use serde::{Deserialize, Serialize};
#[derive(Deserialize, Clone, Debug, Serialize, Default, Display)]
/// ByteFallback is a simple trick which converts tokens looking like `<0x61>`
Expand Down

0 comments on commit ba03c16

Please sign in to comment.