Memo/AmazonWebServices/EMR

EMR(Amazon Elastic MapReduce)

EMR(Amazon Elastic MapReduce) †

Amazon Elastic MapReduce（Hadoopクラスター Amazon EMR） | アマゾンウェブサービス（AWS 日本語）
- Hadoop サービス

Amazon Elastic MapReduce Ruby Client : Developer Tools : Amazon Web Services

ヘルプ

# elastic-mapreduce と同じディレクトリに以下のファイルを作るか、コマンドラインオプションに指定
# access_id, private_key, region 必須
vim credentials.json
----
{
"access_id": "[Your AWS Access Key ID]",
"private_key": "[Your AWS Secret Access Key]",
"keypair": "[Your key pair name]",
"key-pair-file": "[The path and name of your PEM file]",
"log_uri": "[A path to a bucket you own on Amazon S3, such as, s3n://mylog-uri/]",
"region": "[The Region of your job flow, either us-east-1, us-west-2, us-west-1, eu-west-1, ap-northeast-1, ap-southeast-1, or sa-east-1]"
}
----
elastic-mapreduce --help

# JOB一覧はJOB作成したIAMユーザか、AWSアカウントユーザしか取得できない。IAMで権限だけあってもダメ
elastic-mapreduce \
  -c credentials.json \
  --list

aws secret keyに'/'が含まれていると"Invalid hostname in URI"になる場合がある。aws側にあるhadoop coreの問題なのでパッチが当てられない。URIにsecret keyを含まない形で使うか、secret keyを再生成する
- [#HADOOP-3733] "s3:" URLs break when Secret Key contains a slash, even if encoded - ASF JIRA

記事
- Amazon Elastic MapReduceの使い方─Hadoopより手軽にはじめる大規模計算：第1回　Amazon Elastic MapReduceを使う準備をする｜gihyo.jp … 技術評論社

連絡先

TreeView

Memo (711)
- 2FA-MFA
- ACDSee
- ASP
- ASP.NET
- Adobe
- Affiliate
- Aipo
- Akelos (3)
  - sample (2)
    - ajax
    - pref
- Algorithm (1)
  - image
- Aliyun (1)
  - cli
- AmazonWebServices (88)
  - ACM
  - APIGateway
  - Alexa
  - Athena (2)
    - CloudTrail
    - Example
  - Backup
  - Batch
  - Bedrock
  - Billing
  - CDK
  - Chatbot
  - CloudFormation
  - CloudFront
  - CloudSearch
  - CloudTrail
  - CloudWatch
  - CodeBuild
  - CodeCommit
  - CodeDeploy
  - CodePipeline
  - Configservice
  - Connect
  - DataPipeline
  - Detective
  - DirectConnect
  - DirectoryService
  - DocumentDB
  - DynamoDB
  - EC2 (5)
    - AMI
    - DLM
    - EBS
    - ec2-metadata-mock
    - mock-ec2-metadata
  - EC2-classic
  - ECR
  - ECS
  - EKS
  - ELB
  - EMR
  - ElastiCache
  - ElasticBeanstalk
  - Fargate
  - FireLens
  - Glacier
  - Glue
  - GuardDuty
  - Health
  - IAM
  - IPv6
  - IoT
  - KinesisDataFirehose
  - Lambda
  - OpenSearch(Elasticsearch)
  - Organizations
  - QuickSight
  - R53
  - RDS (1)
    - Aurora
  - Route53
  - S3
  - SDB
  - SDK (1)
    - Python
  - SES
  - SNS
  - SSO
  - SecretsManager
  - SystemsManager
  - VPC (1)
    - TransitGateway
  - VPN
  - WAF
  - aws-vault
  - awscli (12)
    - CloudWatch
    - DynamoDB
    - EC2
    - RDS
    - S3
    - backup
    - ce
    - elb
    - error
    - parallel
    - route53
    - v2
- Amazon_Dash_Button
- Amazon_Fire_TV_Stick
- Amazon_Fire_Tablet
- Android (15)
  - AIR
  - ASUS_EeePad_Transformer
  - ASUS_Nexus7
  - ASUS_ZenFone_Max_Pro_M1
  - GALAXY_Tab
  - HTC_Butterfly_s
  - HTC_Desire
  - HTC_Desire_HD
  - HTC_Desire_Z
  - HTC_HT-03A
  - Lenovo_Legion_Y700_2023
  - SHARP_AQUOS_sense4_lite
  - adk
  - app
  - sdk
- Ansible (20)
  - AWS
  - AWX
  - Error
  - Facts
  - Filters
  - Galaxy
  - Install
  - Jinja2
  - Lookups
  - Loops
  - Tower
  - Troubleshooting
  - Validation
  - Variables
  - Vault
  - Windows
  - ldap
  - local_action
  - module_development
  - set_fact
- AppleTV
- Arduino
- ArtificialIntelligence (3)
  - LLM (1)
    - ClaudeCode
  - MachineLearning
- Atlassian
- AugmentedReality
- Azure
- BBS
- Backlog
- Bazaar
- Becky
- Blog
- Blynk
- Browser
- C
- C++
- CDN
- CSS
- ChaosEngineering
- CircleCI
- CloudComputing
- Cloudflare
- ComputerSecurity
- Consul
- Creative
- DNS
- DTM
- Database (1)
  - ツリー(階層)構造の設計
- Datadog
- Docker (4)
  - Dockerfile
  - Kubernetes
  - k3s
  - nomad
- Doxygen
- Dropbox
- DuckDB
- E-book (4)
  - BOOKSCAN
  - Calibre
  - NOOK
  - edit
- EC-CUBE
- EOL
- Electron
- English
- EstimateTechnique
- Evernote
- Excel
- F-PLUG
- F-Secure_PSB
- FFmpeg
- Fabric
- Filer
- Flash (1)
  - Lite
- Flex
- Fonts
- FreeWLAN
- GIS
- Gainer
- Game
- GameDev
- Geolocation
- GlusterFS
- Google (10)
  - AdSense
  - Apps_Script
  - Docs
  - Drive
  - Gemini
  - Gmail
  - NotebookLM
  - SpreadSheet
  - Workspace
  - reCAPTCHA
- GoogleCloudPlatform
- Google_Chromecast
- Gradle
- Grafana (9)
  - API
  - Athena
  - CloudWatch
  - GoogleBigQuery
  - Loki
  - Troubleshooting
  - Zabbix
  - docker
  - plugins
- GraphQL
- Graphics
- Graphviz
- HTML (1)
  - 5
- Heroku (2)
  - Papertrail
  - herokucli
- IPTV
- IPv6
- ImageProcessing
- InfoPath (1)
  - JScript
- IntelliJ_IDEA
- IoT
- JAWBONE_UP
- Java (8)
  - Eclipse (1)
    - Cray
  - Eclipse3.1でiアプリ開発
  - Maven
  - Tomcat
  - log4j
  - 携帯 (1)
    - サンプル
- JavaScript (1)
  - jq
- Jenkins
- JupyterNotebook
- KVM_Switch
- LVGL
- LXC
- LibreOffice
- LifeHacks
- Linux (139)
  - AlmaLinux (2)
    - 10
    - 9
  - AntiVirus
  - Apache
  - Archiver
  - Bash
  - BitTorrent
  - CVS
  - CentOS (5)
    - 5
    - 6
    - 7
    - 8
    - Stream
  - CentOS4
  - Chrony
  - ClamAV
  - Disk
  - Dragonfly
  - Duplicity
  - Fedora core 1
  - Fedora core 5 (2)
    - VMware server
    - samba3.0
  - Firecracker
  - Fluent-Bit
  - Fluentd(td-agent) (1)
    - Errors
  - Heartbeat
  - HyperEstraier
  - InitScript
  - LVM
  - LVS
  - Linuxbrew
  - LiteSpeed
  - Mailman
  - Makefile
  - Memcached
  - Monit
  - Monitoring
  - Munin
  - Namazu
  - OSS-license
  - OpenLDAP (1)
    - WebUI
  - OpenResty
  - Pacemaker
  - Redhat Enterprise Linux ES3
  - Redhat Enterprise Linux ES4
  - SC420 (6)
    - MySQL5.0
    - PostgreSQL8.0
    - Subversion
    - eAccelerator0.9.3
    - php5.0
    - 外部公開(DMZ)
  - SELinux
  - SSL
  - ShellScript
  - Vyatta
  - WebCamera
  - Zabbix (14)
    - 1.8jp
    - 2.0
    - 2.2
    - 2.4
    - 3.0
    - 4.0
    - 5.0
    - 6.0
    - 7.0
    - API
    - AWS
    - Error
    - Template
    - docker
  - ag
  - anyenv
  - audit
  - cloud-init
  - comm
  - command
  - curl
  - datetime
  - denyhosts
  - diff
  - dnf
  - dnsmasq
  - fail2ban
  - find
  - firewalld
  - fzf
  - grep
  - haproxy
  - iptables
  - jailkit
  - jc
  - less
  - lftp
  - logrotate
  - lsof
  - mail
  - mdadm
  - nftables
  - nginx
  - ntp
  - parallel
  - pdsh
  - peco
  - postfix
  - pure-ftpd
  - rclone
  - rdiff-backup
  - redis
  - resolv.conf
  - rootsh
  - rpm
  - rsync
  - samba
  - servicelist
  - socat
  - source-highlight
  - ss
  - ssh
  - sudo
  - symlink
  - syslog
  - systemd
  - taRgrey
  - tcpflow
  - tmux
  - vim
  - vsftpd
  - webdruid
  - wget
  - xargs
  - yum
  - zphoto
- Lua
- MSOffice
- MacOS X (2)
  - Homebrew
  - app
- MariaDB
- Markdown
- MessagePack
- MicrosoftTeams
- MicrosoftVisualStudio
- MongoDB
- Movie
- Mp3tag
- MySQL (5)
  - 5.5
  - 5.7
  - 8.0
  - Error
  - docker
- NETGEAR_ReadyNAS
- NLP
- NetworkNotepad
- NoSQL
- Node.js
- Notion
- OCR
- ODROID
- OSRM
- OSSEC
- OpenOfficeOrg
- OpenTofu
- OpenVPN
- OpenWrt (7)
  - Buffalo_WSR-2533DHP
  - ELECOM_WRC-X3200GST3
  - Package (4)
    - adblock
    - ddns
    - luci
    - openvpn
- Opera
- Oracle
- PC
- PC98
- PDF
- PHP (35)
  - Bug
  - CakePHP
  - Composer
  - Excelファイル生成
  - Google
  - PEAR (11)
    - Auth
    - Cache_Lite
    - DB
    - DB_DataObject
    - File_Archive
    - HTML_QuickForm
    - MDB2
    - QuickForm
    - SQLite
    - Services_Amazon
    - session_pgsql
  - PECL (1)
    - oauth
  - PHP Accelerator
  - SOAP
  - Smarty
  - XML_RPC
  - Xdebug
  - eAccelerator (1)
    - CentOS5.2
  - log4php
  - phpでSSL通信
  - qdmail
  - rss
  - simpletest
  - wkhtmltox
  - xhprof
  - フレームワーク
  - ベンチマーク
- PSP
- PT2 (2)
  - CentOS
  - Ubuntu
- Packer
- Parquet
- PayPal
- Perl (1)
  - OneLiner
- PhantomJS
- Photoshop
- PlantUML
- Poderosa
- Podman
- PostgreSQL (3)
  - constraint
  - pgpool
  - pgpool-II
- PowerPoint
- Programming
- Prometheus
- Puppet (5)
  - Geppetto
  - hiera
  - v3.x
  - v5.x
  - v8
- Python (8)
  - AWS
  - Error
  - Poetry
  - install
  - luigi
  - pip
  - test
  - uv
- QRCode
- RADIUS
- REST_API
- RFID
- RabbitMQ
- Rackspace
- Raspberry_Pi (3)
  - ANAVI_Infrared_pHAT
  - DNSB-35137
  - co2
- RecordingServer
- Redash
- Redmine
- RegularExpression
- Rlogin
- Ruby (1)
  - Rails
- Rundeck
- Rust
- SQL
- SQLite
- SSO
- SakuraEditor
- SendGrid
- Server
- Skype
- Slack
- SoftEther (1)
  - Raspberry_Pi
- Solaris
- Stashboard
- Subversion (6)
  - Install
  - trac (4)
- SwitchBot
- Taskfile
- Tauri
- Terraform (12)
  - Error
  - G_Suite
  - GoogleCloud
  - aliyun
  - aws (1)
    - Errors
  - dynamic-block
  - for_each
  - heroku
  - legacy
  - provider
  - random
- ThinkPad
- Trac
- Traefik
- TrendMicroDeepSecurity
- Twitter
- TypeScript
- USB-IO
- Ubuntu (1)
  - Error
- Unity
- VMware
- VOCALOID
- Vagrant (4)
  - Errors
  - Windows
  - box
  - macOS
- VictoriaMetrics
- VirtualBox
- Visio
- VisualStudioCode (1)
  - AWS
- VxWorks
- WebBrowser (5)
  - Chrome
  - Chromium
  - Firefox (1)
    - PreBar
  - Vivaldi
- Wiki (4)
  - BlockingSpam
  - dev
  - docker
  - 広告
- Windows (28)
  - .NET_Framework
  - 10 (12)
    - WSL (11)
      - 2
      - CentOS
      - Docker
      - Install
      - Ubuntu
      - ssh-agent-wsl
      - wsl-ssh-agent
      - wsl-terminal
      - wsl2-ssh-agent
      - wslgit
      - wsltty
  - 11
  - 7
  - 8
  - ATOK
  - BatchFile
  - EventLog
  - Firewall
  - PackageManagement
  - PowerShell
  - VirtualPC
  - WSA
  - WindowsTerminal
  - winget
  - エラー
- WindowsMobile (3)
  - Link
  - Software
  - ssh
- Wireshark
- Word
- WordPress
- XBMC (2)
  - CentOS
  - Ubuntu
- Xamarin
- YAMAHA-RTX
- antivirus
- backup (1)
  - AcronisTrueImage
- company
- containerd
- csvtsv
- dd-wrt (7)
  - Buffalo_WHR-G301N
  - Buffalo_WXR-1900DHP
  - Buffalo_WZR-1166DHP
  - Buffalo_WZR-1750DHP
  - Buffalo_WZR-D1800H
  - Buffalo_WZR-HP-AG300H
  - Buffalo_WZR-HP-G300NH
- draw.io
- facebook
- favorite
- git (5)
  - Bitbucket
  - GitHub (3)
    - Actions (2)
      - AWS
      - act
- golang
- iPhone (2)
  - app
  - sdk
- iPod
- jQuery
- keepass
- localstack
- mbed
- mise
- mobile
- pandoc
- printer
- programmer
- serverspec
- soracom
- spam
- telework
- twilio
- web
- wkhtmltopdf
- yaml (1)
  - yq
- ユーザビリティ
- 個人情報保護法
- 航空機用語
- 就職
- 設計開発
- 素材集
- 相続
- 投資
- 用語
- FrontPage
  - - instag.inc.php

Memo/AmazonWebServices/EMR

EMR(Amazon Elastic MapReduce) †

Software

Programming

Game

雑記

連絡先

TreeView

人気の10件

最新の10件