Itinerary check: 20 days in October, festivals included?

Hi friends from /JapanTravel!

I am planning a 3-week trip to Japan in October arriving on the 06th, Sunday, a leaving on the 25th, Friday.

This is my first time traveling to Japan (a lifelong dream!) and I will be going with my wife and my cousin. We are in our mid-30s and enjoy walking around a lot, tasting great food and enjoying cultural activities – we are used to traveling and exploring a lot on foot/public transportation.

Our tentative itinerary (as in "places we will be spending a night at") goes like this:

• Tokyo (maybe Hakone)
• Takayama
• Osaka
• Hiroshima
• Mount Koya
• Kyoto
• Tokyo again

And I'd love to get your perspectives on our tentative schedule, please – bolded below are the things about what we have the most questions:

• 06th to 10th: Tokyo – 4 days
  • Arrive on the 06th, and decompress a bit (12h timezone difference!)
  • Activities: TeamLab, walking+eating+shopping at neighborhoods (Shinjuku, Ginza, Akihabara, Roppongi Hills, Odaiba), getting to see the palaces (Imperial Palace), Metropolitan Gvmt Building, etc.
  • 1 or 2 day trips to Kamakura and Kawaguchi Lake/Hakone (possibly spend a night at an onsen around Hakone)
  • Leave to Takayama on the 10th
• 10th to 12th: Takayama – 3 days
  • Hachiman Matsuri on the 10th: still not sure if I should try to attend it or spend an extra night in Tokyo instead?
  • Possible daytrip to Shirakawago
  • Leave to Osaka on the 12th
• 12th to 15th: Osaka – 3 days
  • Kobe/Himeji as an option for a daytrip
  • Activities: Dotonbori, Osaka Castle, Umeda Sky Building, Universal Studios, touring breweries – besides eating, of course!
  • Leave to Hiroshima on the 15th
• 15th to 16th: Hiroshima – 1/2 days
  • Activities: Peace Memorial, Museum of Art, try local cuisine (Hiroshima okonomiyaki, momji manju, carp sashimi, etc)
  • Leave to Mt Koya on the 16th
• 16th to 17th: Mount Koya – 1/2 days
  • Activities: Okunoin cemetery, Kumano Kodo, Kongobu-ji Temple, a bit of hiking if there's time.
  • Spend a night at Mount Koya
  • Leave to Kyoto on the 17th
• 17th to 22nd: Kyoto – 5 days
  • Activities: Fushimi Inari, Kinkaku-ji, Kyoto Imperial Palace, Gion district, Nishiki market, Kiyomizu-dera temple, Arashiyama, Nijo Castle, tea ceremonies if there's time.
  • Possible daytrips to Nara, Mount Kurama
  • Jidai Matsuri and Kurama Fire Festival on the 22nd
  • Leave to Tokyo on the 23rd
• 23rd to 25th: Tokyo – 2 days
  • A bit of unplanned days here: see more of the city, go after anything that might have gone missing (or tips that we might receive throughout our trip) some late shopping and fly back on the 25th.

I had to (unfortunately) deprioritize:

• Kanazawa
• Miyajima
• Nagano

Any feedback is much appreciated! In general, my main questions are related to the 03 festivals:

1. We are making quite the itinerary arrangements (e.g. switching the order of Kyoto <> Osaka and bringing Takayama forward in spite of staying longer in Tokyo) to accommodate Hachiman Matsuri in Takayama and Jidai Matsuri + Kurama Fire Festival in Kyoto – I am afraid that I might be forcing my itinerary too much around those 3 events and I could stay longer in Tokyo as a first time instead (since we also enjoy cities and Tokyo is huge)? Thoughts?
2. Related to #1 above, maybe if we do not prioritize attending those festivals, we might even move faster between cities (e.g. maybe the itinerary is too stranded and commuting in that order to attend the festivals is harder)?

Thank you for your help and guidance – I really appreciate it! 🙂